Mrthdrb

Reading the first amendment strictly through a textualist lens it is understood that Congress cannot limit our speech, but the first amendment does not say anything about companies limiting it, or even the Executive branch limiting our freedom of expression through an executive order.

First Amendment:

Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof; or abridging the freedom of speech, or of the press; or the right of the people peaceably to assemble, and to petition the Government for a redress of grievances.

Since the first amendment text is only about Congress's restrictions on making laws about speech, is the understanding that "we have the freedom of speech" based off of precedent set by courts?

Trivially, yes

The first amendment was adopted on December 15, 1791.

Every time there has been a dispute about what it means that has gone to court since then, the judgement of that court has established, overturned or clarified precedent - that's what common law courts do.

The government can limit your speech

The Supreme Court has recognized categories of speech which receive lesser or no protection from the first amendment. For example, inciting lawless actions, fighting words, true threats, obscenity, child pornography etc.

They have also determined that it doesn't limit the government's power to impose reasonable time, place or manner restrictions on speech. As Justice Holmes put it in Schenck v. United States (1918), "Even the most stringent protection of free speech would not protect a man in falsely shouting fire in a theater and causing panic."

It applies to parts of government which derive their power from Congress

Which is, in most cases, all government.

The executive actually has surprisingly little power granted by the Constitution (Article II, Section 2). All the other powers of the executive are technically delegated powers of Congress and are therefore subject to the first amendment.

Similarly, only the Supreme Court draws its mandate without going through Congress Article III, Section 1) - all other courts are subject to first amendment restrictions.

It only restricts government

The limitation is a negative one on the US Congress (and through incorporation, the states). It does not, of itself, restrict private actors who are free to restrict speech however they want within their own property, including both physical and online spaces.

It is open to the government to enact laws that would extend an affirmative right to free speech onto non-state actors (see Pruneyard Shopping Center v Robins (1980)), however, the Federal government has not done so and neither have most states.

@Alexander noted Marsh v. Alabama in a comment on another answer. I'll expand here as it is relevant to the part of your question that mentions companies.

Briefly, Chickasaw Alabama was a company town where the streets were owned by the company. A Jehovah's Witness tried to distribute leaflets and was arrested for trespass when she refused to stop. The US Supreme Court found in her favour. From the Wikipedia article:

The State attempted to analogize the town's rights to the rights of homeowners to regulate the conduct of guests in their home. The Court rejected that contention, noting that ownership "does not always mean absolute dominion." The court pointed out that the more an owner opens his property up to the public in general, the more his rights are circumscribed by the statutory and constitutional rights of those who are invited in.

In its conclusion, the Court stated that it was essentially weighing the rights of property owners against the rights of citizens to enjoy freedom of press and religion. The Court noted that the rights of citizens under the Bill of Rights occupy a preferred position. Accordingly, the Court held that the property rights of a private entity are not sufficient to justify the restriction of a community of citizens' fundamental rights and liberties.

It will be interesting to see how this plays out in a world where Facebook and Twitter are the equivalent of a privately owned street in a company town.

It appears that you are asking about the idea that the right of free speech in the United States prohibits social media companies from limiting their users' speech. It also appears that you are asking whether this aspect of free speech was established by court precedent. If either of these is incorrect, please advise in a comment.

The answer to that question is no. This aspect of freedom of speech was not established by court precedent, nor was it established by any other means, because it does not exist in US law. This idea of free speech is incorrect, a fundamental misunderstanding. Social media companies, private universities, other private organizations, and private people generally are not required to give others a platform to say whatever they want. On the contrary, they have a right not to be compelled to do so, which is a manifestation of their own right to freedom of speech.

On the other hand, if we look at "the understanding that 'we have the freedom of speech'" as it exists in the US legal system, then the answer to that question is yes, although the reason for that answer is mostly trivial and unsatisfying: any element of the constitutional right of free speech that is not evident from the text of the constitution must necessarily have been established by court precedent.