The Coordination You Cannot See

By: John F Groom

Why a Movie and an Epic Novel Solve Opposite Versions of the Same Problem

A DataUniversa Article

Ask which requires more coordination: a major motion picture or a novel. The answer seems obvious. Consider a film such as Oppenheimer. Hundreds of people appear in the credits. Actors have to be cast and scheduled. Locations must be found, negotiated and prepared. Sets must be designed and built. Costumes have to be created. Cameras, lighting and sound have to work together. Extras have to appear at the correct place and time. Music must be composed or licensed. Contracts have to be negotiated. Insurers and completion guarantors may impose requirements. Thousands of individual shots ultimately have to become one coherent movie.

Now compare that with a novel. At the center of the enterprise is one person sitting at a desk. There may eventually be an agent, publisher, editor, copyeditor, proofreader, cover designer, printer, marketing department, bookseller and distributor. Book publishing involves considerably more coordination than the reader normally sees, but none of this approaches the visible production complexity of a major movie. And yet consider James Clavell’s Shōgun. Suddenly the answer becomes less obvious. The mistake is assuming that coordination occurs only between people.

Two Kinds of Coordination

A large movie presents an extraordinary problem of external coordination. A sprawling novel presents an extraordinary problem of internal coordination. Christopher Nolan could not personally perform every role required to create Oppenheimer. The production required specialists possessing different skills, information and responsibilities to produce one final artifact. The cinematographer’s work had to fit the director’s intentions. The actors’ performances had to fit one another. Sets and costumes had to fit the period. The score had to fit the images. Editing had to fit the performances. Effects had to fit the physical footage.

The essential problem is that many real people must create one coherent imaginary world. The epic novelist faces almost the opposite problem. Clavell had to create a large population of fictional people and keep them coherent himself. The essential problem becomes that one real person must create many coherent imaginary people inhabiting one coherent world. Both are coordination problems. Their complexity simply resides in different places.

The Movie: Coordinating Reality

A movie begins with an unusual disadvantage: reality does not automatically cooperate. Suppose a screenplay says that two characters talk in a seventeenth-century castle at sunset. On the page, that costs almost nothing. On a movie set, it immediately generates dependencies.

There must be a suitable location or constructed set. It must be available. The actors must be available simultaneously. Their costumes and makeup must be ready. Cameras, sound and lighting must be positioned. The weather may matter. Transportation may matter. Labor rules may matter. The production may have only a limited number of hours at the location. And the scene cannot merely be completed. It must fit the rest of the movie.

An actor’s emotional state must make sense relative to a scene that may have been filmed three months earlier. A costume must correspond to the correct point in the story. An object placed on a table in one shot may have to remain there in another shot filmed hours or weeks later. A film therefore contains an enormous number of coordination states.

This is part of what makes a major movie an extraordinary human achievement. Financing alone cannot eliminate the problem. Neither can talent. Money can purchase additional coordination capacity, but it can also create more things that must be coordinated. Add another star and you have added talent, but also another schedule, contract, set of expectations and collection of dependencies. Add another location and you have added visual possibilities, but also transportation, permissions, scheduling and continuity problems. Complexity can create capability while simultaneously creating coordination cost.

The Novel: Coordinating an Imaginary Reality

Now consider Shōgun. Clavell created a sprawling world of political alliances, rivalries, cultural conflicts, religious differences, personal ambitions, military calculations, trade interests and romantic relationships. Characters possess different information. They misunderstand one another. They deceive one another. Their loyalties change. Something established hundreds of pages earlier may constrain what can plausibly happen hundreds of pages later.

The novelist has to remember all of it. A character cannot know something he was never told. A journey cannot take an impossible amount of time. A character established as cautious cannot suddenly behave recklessly simply because the author needs the plot to move forward, unless the author has created a credible reason for the change. A political alliance has consequences, as does a betrayal and a death. The longer the novel becomes, the larger the state space becomes.

And Shōgun adds another remarkable complication. Clavell wrote the novel without having visited Japan. He had substantial experience in Asia, including the terrible experience of being a Japanese prisoner of war during World War II, but the Japan he created for the novel was not a place he had personally walked through and observed.

That matters from a coordination perspective. Direct experience supplies context almost for free. Someone who has spent twenty years in a place does not consciously reconstruct every element of it. Distances, landscapes, buildings, social conventions and ordinary behaviors exist as accumulated mental context. Without that direct experience, context has to be acquired indirectly, reconstructed and then kept consistent with everything else. Clavell therefore wasn’t merely coordinating fictional characters. He was coordinating those characters inside a reconstructed historical world. That is a formidable information problem.

And Then There Is the Reader

There is another constraint that makes the novelist’s task harder still. Everything can fit perfectly and the novel can still fail. The reader has to keep turning the pages.

Information therefore cannot merely be correct. It has to arrive at approximately the correct time. Introduce too many characters too quickly and the reader becomes confused. Explain too much and the story stops moving. Explain too little and events become incomprehensible. Leave a subplot for too long and the reader may forget it. Resolve it too quickly and it may seem trivial.

The author is consequently coordinating not only the fictional world but also the reader’s evolving mental model of that world. At page 100, the reader knows certain things. At page 300, the reader knows more. At page 700, earlier events may acquire completely different significance. The author has to control that progression while allowing the reader to experience it as discovery rather than information management. This is another form of coordination that is almost entirely invisible when it succeeds.

Shōgun Becomes a Movie

Shōgun provides an especially useful example because Clavell’s novel was subsequently transformed for the screen, most famously in the 1980 television miniseries and again in the 2024 FX adaptation. The transformation illustrates what happens when one type of coordination problem becomes another.

Clavell’s internal model could contain enormous amounts of information. A novel can stop for history, thought, explanation and context. It can move directly inside a character’s mind. Hundreds of pages can develop relationships that would require many hours of screen time.

An adaptation has to externalize that world. Now Toranaga is no longer an entity maintained inside an author’s imagination and represented by words. An actor has to portray him. Someone has to create his clothes. Someone has to build or find the environment around him. Other actors have to respond to him. Cameras have to record those interactions. Editors have to select among performances. Language choices have to convey meaning. Thousands of physical details have to collectively persuade an audience that this world exists.

The coordination problem has migrated. The novel concentrates coordination inside a mind. The screen production distributes coordination across an organization. That distinction is much more useful than simply saying that movies are complicated and books are simple.

Coordination Is Not Headcount

This leads to a broader DataUniversa principle: Coordination complexity should not be measured simply by the number of people involved.

Headcount measures one source of coordination difficulty. It does not measure coordination itself. Imagine a task involving 100 people performing identical independent actions. Now imagine one engineer maintaining a software system containing 10,000 interdependent rules. Which has the larger coordination problem? There is no answer until we understand the relationships among the components.

The important variable is not merely how many actors exist. It is how many relevant states, dependencies, meanings and constraints must remain mutually compatible as the system changes over time.

A movie may have high human coordination, temporal coordination, contractual coordination, financial coordination and physical coordination. A multinational business may add legal, jurisdictional, linguistic and cultural coordination. A software platform may have relatively few human participants but extraordinary logical, semantic and technological coordination. An epic novel may have very little organizational coordination while possessing immense narrative, temporal, psychological and semantic coordination. Different systems concentrate coordination in different places.

Shared Context Is a Coordination Technology

The comparison also reveals why shared context is so valuable. Imagine two people who have worked beside each other for twenty years. One can say, “Do it the way we did last time, except reverse the final step.” The other person may immediately understand.

Now give exactly the same instruction to a contractor on another continent who has never worked with the organization. It may be nearly meaningless. The underlying task has not changed. The coordination cost has.

Shared experience functions as compressed information. It allows enormous amounts of context to remain unstated because both parties already possess it. This is one reason coordination becomes harder across countries, organizations, software systems, professional disciplines and cultures. Participants may use exactly the same words without possessing the same underlying context.

And this problem becomes increasingly important as the number of participants increases. Every additional boundary creates another opportunity for context to disappear: human to human, department to department, company to company, database to database, AI to AI, and human to AI. The transmission can succeed perfectly while the coordination fails.

Communication Is Not Coordination

This distinction is particularly important. Modern technology has made communication extraordinarily cheap. We can transmit a document around the world almost instantly. We can hold a video conference with people on five continents. Databases can exchange millions of records. APIs allow software systems to communicate automatically. AI can ingest information from many different sources.

But communication answers only one question: Did the information get there? Coordination asks something much harder: Did the right information reach the right participant, with the right meaning, provenance and context, at the right time, so that the larger system could continue functioning coherently?

A message can be delivered perfectly and still be misunderstood. Two databases can exchange the same field while using different definitions. Two organizations can agree that a person is “active” while applying completely different rules for determining what active means. An AI can combine two individually accurate records and produce an incorrect conclusion because the records were created under different assumptions. The systems communicated. They did not coordinate.

Successful Coordination Becomes Invisible

There is one final similarity between Oppenheimer and Shōgun. When coordination works exceptionally well, the audience stops noticing it. We don’t watch Oppenheimer thinking about transportation schedules, insurance contracts, costume continuity, location agreements, lighting plans and payroll systems. We watch Oppenheimer.

Likewise, we don’t read Shōgun while consciously maintaining a database of characters, relationships, locations, incentives, chronology and information states. We experience Japan. This may be one of the defining characteristics of successful coordination: the machinery disappears into the result.

The reverse is also true. Coordination often becomes visible precisely when it fails. The movie scene somehow feels wrong. A character behaves inconsistently. Software systems disagree about what “complete” means. Two departments believe the other is responsible. A construction project stalls because one approval has not arrived. An AI combines individually valid pieces of information whose meanings are incompatible. People experience the failure at the output. The underlying cause may be coordination.

The DataUniversa Problem

This is where the movie and novel analogy becomes particularly relevant to DataUniversa. Modern information environments increasingly contain both kinds of complexity at once. They resemble the movie because many independent actors have to interact: people, companies, databases, APIs, AI systems, regulators, devices and organizations. But they also resemble the epic novel because enormous numbers of meanings, relationships, histories, authorities and states have to remain coherent.

And unlike Clavell, there may be no single author maintaining the whole thing inside one mind. That is the DataUniversa problem. Connecting the components does not solve it. In fact, connecting more components can make the coordination problem larger.

Interoperability Is Necessary, but It Is Not the End

Interoperability is essential. But interoperability is an enabling condition for coordination, not coordination itself. System A can successfully send information to System B. That establishes technical interoperability. But what if System A and System B attach different meanings to the information? What if one system’s definition changed six months ago? What if one source is authoritative for one purpose but not another? What if two organizations legitimately maintain different definitions because they operate under different rules? What if the information was accurate when created but circumstances subsequently changed?

The systems are interoperable. They are not necessarily coordinated. This is why DataUniversa cannot simply be about making everything conform to a universal schema or a single canonical meaning. Sometimes there is a correct common definition. Sometimes there isn’t. Good coordination may require preserving differences rather than eliminating them. Coordination does not require everything to become the same. It requires different things to remain intelligible to one another.

The Coordination Burden of Connected AI

AI makes this problem considerably more important. Consider an isolated AI answering a question from a single document. Its coordination burden is relatively small. Now connect that AI to ten databases, five external APIs, several specialized models, personal information, company records, real-time information and other AI systems. Its capabilities increase enormously, but so does the number of things that must remain coordinated.

Which source is authoritative? Which definition applies? When was the information true? Under what circumstances was it collected? What does this particular field mean in this particular system? Does another system use the same word differently? What did the AI recommend previously? What action was actually taken? What happened afterward? What did the system learn? What should change the next time a similar situation occurs?

These are not primarily language-generation problems. They are coordination problems. And there is an important paradox here: The more connected AI becomes, the greater its potential intelligence and the greater its coordination burden. A disconnected AI lacks information. A badly coordinated connected AI can possess enormous amounts of information and still reach the wrong conclusion.

Context Is What Makes Coordination Possible

That is why provenance, semantics, time and authority matter so much. Information does not exist independently of context. Who produced it? When? Based upon what evidence? Using what definition? Under what conditions? Who has authority over it? What other information does it depend upon? Has anything changed since it was created? What happened when someone acted upon it?

These questions can appear to be metadata surrounding the “real” information. Often they are what determines what the information actually means. Clavell somehow had to preserve this kind of context mentally. Toranaga knows certain things and does not know others. A relationship has a history. An event occurred before another event. A character’s behavior means something because of what happened 300 pages earlier. Remove enough of that context and Shōgun becomes a collection of sentences rather than a coherent world. Information systems face the same fundamental problem. Data without sufficient context can remain technically accurate while becoming practically misleading.

DataUniversa as a Coordination Layer

This suggests a broader way of understanding DataUniversa. DU can be thought of as a coordination layer. Not simply a layer that moves information. Not a layer that forces every participant to adopt the same definition. Not a layer that assumes every disagreement should be resolved into one canonical answer.

Instead, the objective is to preserve enough of the relationships among data, meaning, provenance, authority, time, evidence, people and action that independently created information can be used together without destroying the distinctions necessary to understand it.

Sometimes that means identifying a common meaning. Sometimes it means preserving competing meanings. Sometimes it means determining which source has authority. Sometimes it means preserving the fact that authority itself depends upon the circumstances. Sometimes it means connecting two things. Sometimes it means knowing that two things should not be collapsed into one. That is coordination.

From Information to Action to Learning

And coordination should not end when information reaches a decision. The real world continues. A person or system takes an action. Something happens. That result becomes new evidence. The evidence may confirm the original expectation or contradict it. That changes what should be believed, and what has been learned should affect what happens next.

So the coordination problem extends across time:

Question → expectation → test → evidence → result → learning → action → next question.

Each stage has to remain connected to what came before it. Otherwise organizations and AI systems can repeatedly encounter the same problem without accumulating meaningful learning. Coordination is therefore not merely about getting different systems to work together at one moment. It is also about keeping past evidence, present understanding and future action connected.

The Movie, the Novel and DataUniversa

We can now return to where we began. The great movie coordinates many real people to create one coherent imaginary world. The great epic novel coordinates many imaginary people inside one real person’s mind to create one coherent world. One distributes complexity. The other concentrates it. DataUniversa confronts a third version of the same problem. The modern information environment contains many people, many organizations, many databases, many definitions, many AI systems and many histories. There is no single director controlling all of them. There is no single novelist holding the entire world inside one mind. And we should not expect all of those independent systems to become identical.

The challenge is to allow them to remain different while preserving enough context that they can nevertheless function coherently together. That is why coordination is a larger concept than communication. It is larger than connectivity. And ultimately it is larger than interoperability. The novelist preserves coherence mentally. The filmmaker preserves coherence organizationally. DataUniversa seeks to preserve coherence computationally. The objects being coordinated are different, but the fundamental problem is the same: Things that remain different must nevertheless continue to make sense together. That may be the central coordination problem of connected AI and one of the central problems DataUniversa is being built to solve.