Can a self-taught enthusiast in an unrelated occupation, without a law degree, without any sort of formal legal education nor bar admittance, help other people study law? What would be the limitations?
An enthusiast can discuss the law in general terms without applying laws to particular facts, without claiming any legal credential, and without making recommendations as to a course of action.
For example, an economics professor can discuss how various rules of law impact the economy in general, and a PhD in political science, or a social studies or journalism teacher, could discuss constitutional law in general.
An essential element of the practice of law is either acting on behalf of someone else, or providing fact specific advice to someone else. Circumstances that involve neither of these circumstances are almost never the practice of law. Circumstances that involve one or the other are often, but not always, the practice of law.
There is considerable regional variation in what constitutes the practice of law. In general, the practice of law is construed more narrowly in the West (including both Texas and California) than in the East, and more narrowly in historically sparsely populated states (e.g. Alaska and Montana or Colorado or Wyoming), than in more densely populated states (including both Texas and California). Thus, California and Texas tend to be in the middle of the range from very narrow definitions of the practice of law to very expansive definitions of the practice of law.
Basically, places with more lawyers per capita historically have done more to protect lawyer's turf, while in places that have historically had fewer lawyer's per capita, conduct that could arguably be considered the practice of law has been tolerated out of necessity.
Also, what constitutes the practice of law in connection with the federal government is often more lenient than what constitutes the practice of law in connection with the state government.
For example, administrative law forums related to the federal government (e.g. tax court and representing clients in audits, or government contract disputes) more often permit non-lawyers to conduct legal practice-like activity than administrative law forums in state government, although sometimes non-lawyer participation is allowed in both (e.g. non-lawyers are often permitted in unemployment hearings).
For simplicity, let's assume an honest person who makes it clear that they're not a lawyer: can they volunteer as a first step in informing another person of their possible legal options (e.g., in landlord-tenant disputes), such that a person is more knowledgeable in case they do decide to contact or retain a lawyer?
This isn't perfectly cut and dry, but basically once someone is applying the law to the facts of someone else's case, or making recommendations on a course of action (other than to merely get a lawyer in a particular situation), no.
It is acceptable to inform someone of places where low cost or free legal advice or self-help resources can be obtained, for example, in a directory type format.
Can such an enthusiast accept fees and/or gifts directly from someone they help study law, like a beer or lunch, for their help/suggestions in navigating the law? Can they casually mention that if the person wins their case, and thinks that such an enthusiast helped them do so, that a payment would be welcome? What would be the limitations?
No, other than minimal payment for being a scrivener (i.e. basically typist or interpreter) as described below, or for publications or lectures on general legal topics as described above.
So, can a volunteer enthusiast help represent someone else in a small claims court, in either California or Texas?
No. This is a cut and dry rule. The only exception is the corporate officers or LLC managers can usually represent an entity in small claims court.
If an enthusiast helps prepare certain documents on behalf of the plaintiff for Small Claims Court in California or Texas,
A non-lawyer can help someone who is illiterate or doesn't speak English or is functionally illiterate or can't type fill in official forms and directing people to the relevant instructions, and can review court forms to see that all blanks were completed and that the answers are in the right fill in the blank parts of the form (e.g. which person is the plaintiff and which person is the defendant). But, otherwise no.
for example, even the demand letter that would have to be sent before a small claims action could be started in Texas, could they negotiate with the plaintiff for a certain payout of the money recovered?
No and no, unless the demand letter was written by the person making the demand on their own behalf. A limited exception is usually recognized for insurance adjusters evaluating claims on behalf of an insurance company or pre-litigation resolution of disputes by collection agencies and employees of creditors.