|
| 1 | +--- |
| 2 | +slug: fluxninja-joins-coderabbit |
| 3 | +title: FluxNinja joins CodeRabbit |
| 4 | +description: |
| 5 | + CodeRabbit has acquired FluxNinja, a startup that provides a platform for |
| 6 | + building scalable generative AI applications. |
| 7 | +authors: [gur, vishu] |
| 8 | +tags: |
| 9 | + [ |
| 10 | + "AI", |
| 11 | + "Acquisition", |
| 12 | + "Generative AI", |
| 13 | + "Code Review", |
| 14 | + "Load Management", |
| 15 | + "Rate Limiting", |
| 16 | + "Caching", |
| 17 | + "Scheduler", |
| 18 | + ] |
| 19 | +image: ./preview.png |
| 20 | +--- |
| 21 | + |
| 22 | + |
| 23 | + |
| 24 | +We are excited to announce that CodeRabbit has acquired |
| 25 | +[FluxNinja](https://fluxninja.com), a startup that provides a platform for |
| 26 | +building scalable generative AI applications. This acquisition will allow us to |
| 27 | +ship new use cases at an industrial-pace while sustaining our rapidly growing |
| 28 | +user base. FluxNinja's Aperture product provides advanced rate & concurrency |
| 29 | +limiting, caching, and request prioritization capabilities that are essential |
| 30 | +for reliable and cost-effective AI workflows. |
| 31 | + |
| 32 | +<!--truncate--> |
| 33 | + |
| 34 | +Since our launch, |
| 35 | +[Aperture's open-source](https://github.com/fluxninja/aperture) core engine has |
| 36 | +been critical to our infrastructure. Our initial use case centered around |
| 37 | +[mitigating aggressive rate limits](../openai-rate-limits-2023-10-23/blog.md) |
| 38 | +imposed by OpenAI, allowing us to prioritize paid and real-time chat users |
| 39 | +during peak load hours while queuing requests from the free users. Further, we |
| 40 | +used Aperture's |
| 41 | +[caching and rate-limiting capabilities](../how-we-built-cost-effective-generative-ai-application-2023-12-23/blog.md) |
| 42 | +to manage costs that in turn allowed us to offer open-source developers a fully |
| 43 | +featured free tier by minimizing abuse. These capabilities allowed us to scale |
| 44 | +our user base without ever putting up a waitlist and at a price point that is |
| 45 | +sustainable for us. With Aperture's help, CodeRabbit has scaled to over 100K |
| 46 | +repositories and several thousand organizations under its review in a short |
| 47 | +period. |
| 48 | + |
| 49 | +We started CodeRabbit with a vision to build an AI-first developer tooling |
| 50 | +company from the ground up. Building enterprise-ready applied AI tech is unlike |
| 51 | +any other software engineering challenge of the past. Based on our learnings |
| 52 | +while building complex workflows, it became apparent that we need to invest in a |
| 53 | +platform that can solve the following problems: |
| 54 | + |
| 55 | +- Prompt rendering: Prompt design and rendering is akin to responsive web |
| 56 | + design. Web servers render pages based on the screen size and other |
| 57 | + parameters, for example, on a mobile device, navigation bars are usually |
| 58 | + rendered as hamburger menus, making it easier for human consumption. |
| 59 | + Similarly, we need a prompt server that can render prompts based on the |
| 60 | + context windows of underlying models and prioritize the packing of context |
| 61 | + based on business attributes, making it easier for AI consumption. It's not |
| 62 | + feasible to include the entire repository, past conversations, documentation, |
| 63 | + learnings, etc. in a single code review prompt because of the context window |
| 64 | + size limitations. Even if it was possible, AI models exhibit poor recall when |
| 65 | + doing an inference on a completely packed context window. While tight packing |
| 66 | + may be acceptable for use cases like chat, it’s not for use cases like code |
| 67 | + reviews that require accurate inferences. Therefore, it's critical to render |
| 68 | + prompts in such a way that the quality of inference is high for each use-case, |
| 69 | + while being cost-effective and fast. In addition to packing logic, basic |
| 70 | + guardrails are also needed, especially when rendering prompts based on inputs |
| 71 | + from end-users. Since we provide a free service to public repositories, we |
| 72 | + have to ensure that our product is not misused beyond its intended purpose or |
| 73 | + tricked into divulging sensitive information, which could include our base |
| 74 | + prompts. |
| 75 | + |
| 76 | +- Validation & quality checks: Generative AI models consume text and output |
| 77 | + text. On the other hand, traditional code and APIs required structured data. |
| 78 | + Therefore, the prompt service needs to expose a RESTful or gRPC API that can |
| 79 | + be consumed by the other services in the workflow. We touched upon the |
| 80 | + rendering of prompts based on structured requests in the previous point, but |
| 81 | + the prompt service also needs to parse, validate responses into structured |
| 82 | + data and measure the quality of the inference. This is a non-trivial problem, |
| 83 | + and multiple tries are often required to ensure that the response is thorough |
| 84 | + and meets the quality bar. For instance, we found that when we pack multiple |
| 85 | + files in a single code review prompt, AI models often miss hunks within a file |
| 86 | + or miss files altogether, leading to incomplete reviews. |
| 87 | + |
| 88 | +- Observability: One key challenge with generative AI and prompting is that it's |
| 89 | + inherently non-deterministic. The same prompt can result in vastly different |
| 90 | + outputs, which can be frustrating, but this is precisely what makes AI systems |
| 91 | + powerful in the first place. Even slight variations in the prompt can result |
| 92 | + in vastly inferior or noisy outputs, leading to a decline in user engagement. |
| 93 | + At the same time, the underlying AI models are ever-evolving, and the |
| 94 | + established prompts drift over time as the models get regular updates. |
| 95 | + Traditional observability is of little use here, and we need to rethink how we |
| 96 | + classify and track generated output and measure quality. Again, this is a |
| 97 | + problem that we have to solve in-house. |
| 98 | + |
| 99 | +While FluxNinja's Aperture project was limited to solving a different problem |
| 100 | +around load management and reliability, we found that the underlying technology |
| 101 | +and the team's expertise were a perfect foundation for building the AI platform. |
| 102 | +Prompt engineering is in its nascent stage but is emerging as a joystick for |
| 103 | +controlling AI behavior. Packing the context window with relevant documents |
| 104 | +(retrieval augmented generation, aka RAG) is also emerging as the preferred way |
| 105 | +of providing proprietary data compared to fine-tuning the model. Most AI labs |
| 106 | +focus on increasing the context window rather than making fine-tuning easier or |
| 107 | +cheaper. Despite the emergence of these clear trends, applied AI systems are |
| 108 | +still in their infancy. None of the recent AI vendors seem to be building the |
| 109 | +"right" platform, as most of their focus has been on background/durable |
| 110 | +execution frameworks, model routing proxies/gateways, composable RAG pipelines, |
| 111 | +and so on. Most of these approaches fall short of what a real-world AI workflow |
| 112 | +requires. The right abstractions and best practices will still have to appear, |
| 113 | +and the practitioners themselves will have to build them. AI platforms will be a |
| 114 | +differentiator for AI-first companies, and we are excited to tackle this problem |
| 115 | +head-on with a systems engineering mindset. |
| 116 | + |
| 117 | +We are excited to have the FluxNinja team on board and to bring our users the |
| 118 | +best-in-class AI workflows. We are also happy to welcome |
| 119 | +[Harjot Gill](https://www.linkedin.com/in/harjotsgill/), the founder of |
| 120 | +FluxNinja, and the rest of the team to CodeRabbit. |
0 commit comments