Your diagnosis is well-aligned with known issues in large Convex projects. The “Type instantiation is excessively deep and possibly infinite” (TS2589) error is a recognized scaling pain point, especially as the number of Convex functions grows and the aggregate complexity of the generated api type increases. This is not unique to your project—multiple users have reported that as their codebase and the number of Convex functions grow, TypeScript’s type inference and checking can become prohibitively slow or even break entirely, particularly when importing the generated api object into frontend code. This is due to the large, deeply nested type graphs that TypeScript must resolve for the api object, which can overwhelm the compiler at a certain scale Typescript performance issues with caused by Convex types (Discord).
Explicit Return Types
Your plan to add explicit Promise<ReturnType> annotations to every handler is strongly supported by both Convex staff and the community as a mitigation strategy. The lack of explicit return types can cause TypeScript to attempt to infer types across the entire dependency graph, which quickly becomes intractable as the number of functions grows. By explicitly annotating return types, you help TypeScript “short-circuit” this inference, reducing the complexity of the generated types and improving both performance and reliability Dealing with circular type inference (Convex docs), TS2589: Type instantiation is excessively deep and possibly infinite (Discord).
“Any circular query reference needs to have a handler return type. Example:

export const myAction = action({
  args: {},
  handler: async (ctx): Promise<null> => {
    const result = await ctx.runQuery(api.myFunctions.getSomething);
    return result;
  },
});
TypeScript will check that the type annotation matches what the called query or mutation returns, so you don't lose any type safety.”
— Convex docs
Additional Strategies
Minimize Type Complexity: Where possible, use minimal types for handler return values, especially if only a subset of properties is needed. Avoid returning large, deeply nested objects if not necessary TS2589: Type instantiation is excessively deep and possibly infinite (Discord).
Break Up Large Files: Splitting large files and reducing the number of imports per file can help TypeScript’s performance Typescript performance issues with caused by Convex types (Discord).
Static Code Generation (Beta): Convex is experimenting with static code generation, which can dramatically reduce type complexity by generating static types for the API and data model. This is already being used by some large customers and may be the “endgame” for very large projects. You can enable this in your convex.json (even if the schema doesn’t recognize the field, it should still work if the build passes) Typescript performance issues with caused by Convex types (Discord).
Monitor for Circular Dependencies: Use tools like madge to detect and resolve circular dependencies, which can exacerbate type complexity issues.
Upcoming Features: There is community and internal discussion about features like generating a “light” client, per-function exports, or allowing exclusion lists in the Convex config to limit what gets included in the generated API type. These are not yet available, but are recognized as desirable for large projects Code Spelunking: Uncovering Convex's API Generation Secrets.
Summary
Your diagnosis is correct: the aggregate complexity of the generated api type is the root cause.
Adding explicit return types to all handlers is the recommended and most effective mitigation.
Consider static code generation and keep an eye on future Convex features for large projects.
Continue using your current workaround (const api: any = require(...)) as a temporary measure, but aim to remove it once your type complexity is reduced.
If you run into issues with static code generation or want to discuss large-scale best practices, the Convex team is actively interested in feedback from large customers like you.