When people talk about “web APIs,” they often jump straight to REST and JSON. But the Web’s first practical interfaces—born between roughly 1990 and 1994—were already API-like. They were simple, text-based, and sometimes informal, yet they enabled automated tool use in a way that would feel familiar to modern developers building agentic workflows.

This chapter focuses on a specific thread in early web API history: agents. Not AI agents in today’s sense, but automated clients—scripts, crawlers, gateways, and programs—built to fetch, transform, index, and monitor web resources over HTTP. In the early Web, the boundary between “browser,” “tool,” and “client library” was thin. That thin boundary is where web APIs began.

1990–1991: HTTP starts as a minimal remote procedure for documents

The Web began as a way to retrieve linked documents. In that earliest period, the HTTP interaction pattern was intentionally small: a client connects to a server, requests a resource, and receives a response. Even in its simplest form, that’s an API contract: a defined request structure and a predictable response.

Early HTTP implementations (often described historically as “HTTP/0.9”) were minimal—primarily a GET for a document, with a response that was essentially the document content. The important point for API history isn’t that it lacked headers or formal error handling (those arrived as the protocol matured), but that it established an automatable interface with stable semantics:

• Uniform resource identifiers meant programs could store and reuse addresses.
• Client/server separation meant tools could act remotely without UI automation.
• Stateless requests (in practice) made scripting and batch jobs straightforward.

Those three properties—addressability, remote access, and repeatability—are the DNA of web APIs. Even before the Web became mainstream, they made it possible to write “agent-like” tools that could retrieve information on schedule, parse it, and use it elsewhere.

1992–1993: The rise of “user agents” and programmable clients

As the Web expanded beyond a single research environment, it attracted multiple client programs: early browsers, command-line fetchers, and library-based clients used by developers. The phrase user agent emerged to describe the software acting on behalf of a user. That concept is easy to overlook, but it’s crucial to agent-style automation: the Web began to treat the client as an identifiable actor with behavior that servers might adapt to.

During this period, HTTP moved toward richer request/response structures, including headers. One header that became culturally and technically significant was User-Agent. Even when not strictly required, it gave servers a way to log, analyze, and sometimes tailor responses based on the calling software. In modern API terms, it’s an early form of “client identity.”

At the same time, tool-building accelerated because developers could reuse code. The W3C’s early libwww library (commonly referenced in Web histories) represented an important shift: instead of treating HTTP as something only a browser does, it became a general-purpose network capability you could embed into other programs. That’s a key milestone in web API history: HTTP became an interface not only for humans, but for other software components.

Once a library exists, agents follow. You can fetch resources in bulk, crawl a set of pages to build an index, or retrieve content to feed another system. This wasn’t “API-first,” but it was automation-first—a pattern we still see whenever developers repurpose an existing web interface for programmatic use.

Gateways: early “API adapters” between the Web and older systems

Between 1990 and 1994, the Web wasn’t the only networked information system around. Gopher, WAIS, FTP archives, and other repositories existed and had their own access methods. A practical way to grow the Web quickly was to build gateways: server-side programs that translated a web request into a query against another system, then translated the results back into something web clients could display.

Gateways matter to API history because they look like modern API mediation:

• The Web endpoint provides a stable URL interface.
• A backend integration performs data retrieval and transformation.
• The response is normalized for the calling client.

In modern terms, a gateway is an adapter pattern: “HTTP in front, something else behind.” Between 1990 and 1994, that approach helped turn the Web into a universal access layer. It also reinforced an idea that would later become central to web APIs: HTTP is not just a document protocol; it’s a general integration surface.

1993–1994: CGI turns web servers into programmable interfaces

If early HTTP made content retrievable, the Common Gateway Interface (CGI)—developed around the early-to-mid 1990s—made content generatable. With CGI, a web server could execute an external program and return its output as the HTTP response. This was a major step toward dynamic endpoints, where the returned representation depends on inputs.

CGI’s most historically important contribution to web API evolution was the normalization of request data in a way programs could consume:

• Query strings (key/value parameters) for GET-style inputs.
• Form submissions enabling structured user input.
• Environment variables providing method, path, content type, and more.

Although CGI responses were commonly HTML, the shape of the interaction is recognizably API-like: parameters in, computation happens, data out. You could build a searchable directory, a status endpoint, or a report generator. And crucially, those endpoints could be called by automated tools—not only by humans using browsers.

This is where “agent APIs” begin to sound less metaphorical. Even without a formal API specification, a team could create a stable URL with stable parameters and use it from scripts. That is a proto-API contract—an agreement enforced by convention, not by an OpenAPI document.

Early automated agents: crawling, indexing, and the first etiquette rules

As the Web became more useful, automated access scaled up. Indexing required crawlers—programs that systematically fetched pages and followed links. Some early crawlers are often cited in Web histories as emerging in the early 1990s, and their existence created a new operational reality: servers weren’t just serving humans; they were serving machines at machine speed.

Once that happened, two “API governance” problems appeared almost immediately:

1. Load management: uncontrolled crawling could overload servers.
2. Intent signaling: site owners needed a way to express what automated agents should or should not fetch.

By 1994, the community began converging on a lightweight convention that addressed both issues: robots.txt, a simple text file used to communicate crawling rules. It wasn’t a formal standard at first, but it functioned like one because it created a predictable control surface for automated clients.

From a web API history perspective, robots.txt is fascinating because it’s an early example of an “API for agents.” It acknowledges that non-human clients are first-class participants on the Web and creates a minimal interface for governing them.

Why this era matters for today’s agentic tool use

Modern “agents” are often described as systems that choose tools, call APIs, and chain actions. If you squint, the 1990–1994 Web already supported a simpler version of that loop:

• Discovery: links and URLs provided a graph of callable resources.
• Invocation: HTTP requests executed retrieval or triggered server-side programs (via gateways and CGI).
• Interpretation: clients parsed responses (often HTML) to extract data or follow new links.
• Policy: conventions like User-Agent and robots.txt began to shape automated behavior.

That loop is a direct ancestor of today’s automated tool use. The difference is not the existence of automation, but the maturity of contracts: we now have explicit schemas, auth standards, rate limits, and observability—whereas early Web automation was mostly convention and caution.

If you’re building automation today, it’s worth remembering this lineage. Many of the best practices we treat as modern—identifying your client, respecting site policies, keeping requests idempotent when possible—started as social and technical responses to early web automation.

For more modern explorations of automation patterns and tool-oriented workflows, you can also browse resources at Automated Hacks.

A practical takeaway: early HTTP “interfaces” taught the Web to be programmable

In 1990–1994, the Web’s interface surface was not called an API, but it behaved like one. The emergence of headers, the identification of user agents, the spread of client libraries, the use of gateways, and the programmability unlocked by CGI all pushed the Web from “hypertext documents” toward “callable services.”

If you want a modern, authoritative reference for how HTTP works today (methods, headers, semantics), MDN’s documentation is a solid place to start: https://developer.mozilla.org/en-US/docs/Web/HTTP. Reading it with early Web history in mind makes the evolution of web APIs feel less like a sudden invention and more like a steady expansion of the same basic contract.

FAQ

Were there “web APIs” in 1990–1994?

Not in the modern sense of formally documented JSON endpoints, but there were stable HTTP interfaces that software could call. Gateways and CGI endpoints often behaved like APIs: parameters in, computed output out, accessed over URLs.

What did “agent” mean on the early Web?

It typically meant a client program acting on behalf of a user (a “user agent”) or an automated program like a crawler. The term helped distinguish the calling software from the server and encouraged conventions for identification and responsible automation.

Why is robots.txt part of web API history?

Because it created a simple, predictable interface specifically for automated clients. It’s an early mechanism for machine-to-machine governance on the Web—an “API” that tells agents how to behave.

How did CGI influence later API design?

CGI normalized the idea that a URL can represent a program, not just a file. It established patterns for passing inputs (query strings, form data) and producing dynamic responses—concepts that later frameworks and API architectures refined.