Skip to content

Commit 3160f87

Browse files
shortcutsshortcuts
authored
fix(clients): reduce chances of Push rate limiting (#5153)
1 parent e3c9fd4 commit 3160f87

File tree

13 files changed

+270
-182
lines changed

13 files changed

+270
-182
lines changed

‎clients/algoliasearch-client-java/.java-version

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
21.0.8
1+
21.0.6

‎clients/algoliasearch-client-php/lib/FormDataProcessor.php

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@
1212
*/
1313

1414
/**
15-
* Search API.
15+
* Ingestion API.
1616
*
17-
* The Algolia Search API lets you search, configure, and manage your indices and records. ## Client libraries Use Algolia's API clients and libraries to reliably integrate Algolia's APIs with your apps. The official API clients are covered by Algolia's [Service Level Agreement](https://www.algolia.com/policies/sla/). See: [Algolia's ecosystem](https://www.algolia.com/doc/guides/getting-started/how-algolia-works/in-depth/ecosystem/) ## Base URLs The base URLs for requests to the Search API are: - `https://{APPLICATION_ID}.algolia.net` - `https://{APPLICATION_ID}-dsn.algolia.net`. If your subscription includes a [Distributed Search Network](https://dashboard.algolia.com/infra), this ensures that requests are sent to servers closest to users. Both URLs provide high availability by distributing requests with load balancing. **All requests must use HTTPS.** ## Retry strategy To guarantee high availability, implement a retry strategy for all API requests using the URLs of your servers as fallbacks: - `https://{APPLICATION_ID}-1.algolianet.com` - `https://{APPLICATION_ID}-2.algolianet.com` - `https://{APPLICATION_ID}-3.algolianet.com` These URLs use a different DNS provider than the primary URLs. You should randomize this list to ensure an even load across the three servers. All Algolia API clients implement this retry strategy. ## Authentication To authenticate your API requests, add these headers: - `x-algolia-application-id`. Your Algolia application ID. - `x-algolia-api-key`. An API key with the necessary permissions to make the request. The required access control list (ACL) to make a request is listed in each endpoint's reference. You can find your application ID and API key in the [Algolia dashboard](https://dashboard.algolia.com/account). ## Request format Depending on the endpoint, request bodies are either JSON objects or arrays of JSON objects, ## Parameters Parameters are passed as query parameters for GET and DELETE requests, and in the request body for POST and PUT requests. Query parameters must be [URL-encoded](https://developer.mozilla.org/en-US/docs/Glossary/Percent-encoding). Non-ASCII characters must be UTF-8 encoded. Plus characters (`+`) are interpreted as spaces. Arrays as query parameters must be one of: - A comma-separated string: `attributesToRetrieve=title,description` - A URL-encoded JSON array: `attributesToRetrieve=%5B%22title%22,%22description%22%D` ## Response status and errors The Search API returns JSON responses. Since JSON doesn't guarantee any specific ordering, don't rely on the order of attributes in the API response. Successful responses return a `2xx` status. Client errors return a `4xx` status. Server errors are indicated by a `5xx` status. Error responses have a `message` property with more information. ## Version The current version of the Search API is version 1, as indicated by the `/1/` in each endpoint's URL.
17+
* The Ingestion API lets you connect third-party services and platforms with Algolia and schedule tasks to ingest your data. The Ingestion API powers the no-code [data connectors](https://dashboard.algolia.com/connectors). ## Base URLs The base URLs for requests to the Ingestion API are: - `https://data.us.algolia.com` - `https://data.eu.algolia.com` Use the URL that matches your [analytics region](https://dashboard.algolia.com/account/infrastructure/analytics). **All requests must use HTTPS.** ## Authentication To authenticate your API requests, add these headers: - `x-algolia-application-id`. Your Algolia application ID. - `x-algolia-api-key`. An API key with the necessary permissions to make the request. The required access control list (ACL) to make a request is listed in each endpoint's reference. You can find your application ID and API key in the [Algolia dashboard](https://dashboard.algolia.com/account). ## Request format Request bodies must be JSON objects. ## Response status and errors Response bodies are JSON objects. Successful responses return a `2xx` status. Client errors return a `4xx` status. Server errors are indicated by a `5xx` status. Error responses have a `message` property with more information. ## Version The current version of the Ingestion API is version 1, as indicated by the `/1/` in each endpoint's URL.
1818
*
1919
* The version of the OpenAPI document: 1.0.0
2020
* Generated by: https://openapi-generator.tech
@@ -29,7 +29,7 @@
2929

3030
namespace Algolia\AlgoliaSearch;
3131

32-
use Algolia\AlgoliaSearch\Model\Search\ModelInterface;
32+
use Algolia\AlgoliaSearch\Model\Ingestion\ModelInterface;
3333
use DateTime;
3434
use GuzzleHttp\Psr7\Utils;
3535
use Psr\Http\Message\StreamInterface;

‎playground/javascript/node/algoliasearch.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,7 @@ async function testAlgoliasearchBridgeIngestion() {
152152
console.log('replaceAllObjectsWithTransformation', await client.replaceAllObjectsWithTransformation({
153153
indexName: 'boyd',
154154
objects: [{ objectID: 'foo', data: { baz: 'baz', win: 42 } }, { objectID: 'bar', data: { baz: 'baz', win: 24 } }],
155-
batchSize: 2
155+
batchSize: 1
156156
}));
157157
}
158158

‎scripts/cts/testServer/replaceAllObjectsWithTransformation.ts

Lines changed: 49 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,9 @@ const raowtState: Record<
1010
string,
1111
{
1212
copyCount: number;
13+
deleteCount: number;
1314
pushCount: number;
15+
getEventCount: number;
1416
tmpIndexName: string;
1517
waitTaskCount: number;
1618
waitingForFinalWaitTask: boolean;
@@ -33,6 +35,28 @@ function addRoutes(app: Express): void {
3335
}),
3436
);
3537

38+
app.delete('/1/indexes/:indexName', (req, res) => {
39+
expect(req.params.indexName).to.match(/^cts_e2e_replace_all_objects_with_transformation_(.*)$/);
40+
41+
const lang = req.params.indexName.replace('cts_e2e_replace_all_objects_with_transformation_', '');
42+
if (!raowtState[lang] || raowtState[lang].successful) {
43+
raowtState[lang] = {
44+
copyCount: 0,
45+
pushCount: 0,
46+
getEventCount: 0,
47+
deleteCount: 1,
48+
waitTaskCount: 0,
49+
tmpIndexName: req.params.indexName,
50+
waitingForFinalWaitTask: false,
51+
successful: false,
52+
};
53+
} else {
54+
raowtState[lang].deleteCount++;
55+
}
56+
57+
res.json({ taskID: 123 + raowtState[lang].copyCount, deletedAt: '2021-01-01T00:00:00.000Z' });
58+
});
59+
3660
app.post('/1/indexes/:indexName/operation', (req, res) => {
3761
expect(req.params.indexName).to.match(/^cts_e2e_replace_all_objects_with_transformation_(.*)$/);
3862

@@ -47,6 +71,8 @@ function addRoutes(app: Express): void {
4771
raowtState[lang] = {
4872
copyCount: 1,
4973
pushCount: 0,
74+
getEventCount: 0,
75+
deleteCount: 0,
5076
waitTaskCount: 0,
5177
tmpIndexName: req.body.destination,
5278
waitingForFinalWaitTask: false,
@@ -62,9 +88,12 @@ function addRoutes(app: Express): void {
6288
case 'move': {
6389
const lang = req.body.destination.replace('cts_e2e_replace_all_objects_with_transformation_', '');
6490
expect(raowtState).to.include.keys(lang);
91+
console.log(raowtState[lang]);
6592
expect(raowtState[lang]).to.deep.equal({
6693
copyCount: 2,
67-
pushCount: 10,
94+
pushCount: 4,
95+
getEventCount: 4,
96+
deleteCount: 0,
6897
waitTaskCount: 2,
6998
tmpIndexName: req.params.indexName,
7099
waitingForFinalWaitTask: false,
@@ -94,21 +123,25 @@ function addRoutes(app: Express): void {
94123
expect(req.body.action === 'addObject').to.equal(true);
95124
expect(req.query.referenceIndexName === `cts_e2e_replace_all_objects_with_transformation_${lang}`).to.equal(true);
96125

97-
raowtState[lang].pushCount += req.body.records.length;
126+
raowtState[lang].pushCount++;
98127

99128
res.json({
100-
runID: 'b1b7a982-524c-40d2-bb7f-48aab075abda',
129+
runID: `b1b7a982-524c-40d2-bb7f-48aab075abda_${lang}`,
101130
eventID: `113b2068-6337-4c85-b5c2-e7b213d8292${raowtState[lang].pushCount}`,
102131
message: 'OK',
103132
createdAt: '2022-05-12T06:24:30.049Z',
104133
});
105134
});
106135

107136
app.get('/1/runs/:runID/events/:eventID', (req, res) => {
137+
const lang = req.params.runID.match(/^b1b7a982-524c-40d2-bb7f-48aab075abda_(.*)$/)?.[1] as string;
138+
139+
raowtState[lang].getEventCount++;
140+
108141
res.json({
109142
status: 'succeeded',
110-
eventID: '113b2068-6337-4c85-b5c2-e7b213d82921',
111-
runID: 'b1b7a982-524c-40d2-bb7f-48aab075abda',
143+
eventID: req.params.eventID,
144+
runID: req.params.runID,
112145
type: 'fetch',
113146
batchSize: 1,
114147
publishedAt: '2022-05-12T06:24:30.049Z',
@@ -123,10 +156,18 @@ function addRoutes(app: Express): void {
123156

124157
raowtState[lang].waitTaskCount++;
125158
if (raowtState[lang].waitingForFinalWaitTask) {
126-
expect(req.params.taskID).to.equal('777');
127-
expect(raowtState[lang].waitTaskCount).to.equal(3);
128-
129159
raowtState[lang].successful = true;
160+
expect(req.params.taskID).to.equal('777');
161+
expect(raowtState[lang]).to.deep.equal({
162+
copyCount: 2,
163+
pushCount: 4,
164+
getEventCount: 4,
165+
deleteCount: 0,
166+
waitTaskCount: 3,
167+
tmpIndexName: req.params.indexName,
168+
waitingForFinalWaitTask: true,
169+
successful: true,
170+
});
130171
}
131172

132173
res.json({ status: 'published', updatedAt: '2021-01-01T00:00:00.000Z' });

‎templates/go/ingestion_helpers.mustache

Lines changed: 40 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,12 @@ func (c *APIClient) ChunkedPush(indexName string, objects []map[string]any, acti
1616
batchSize: 1000,
1717
}
1818

19+
offset := 0
20+
waitBatchSize := conf.batchSize / 10
21+
if waitBatchSize < 1 {
22+
waitBatchSize = conf.batchSize
23+
}
24+
1925
for _, opt := range opts {
2026
opt.apply(&conf)
2127
}
@@ -58,31 +64,42 @@ func (c *APIClient) ChunkedPush(indexName string, objects []map[string]any, acti
5864
responses = append(responses, *resp)
5965
records = make([]map[string]any, 0, len(objects)%conf.batchSize)
6066
}
61-
}
6267

63-
if conf.waitForTasks {
64-
for _, resp := range responses {
65-
_, err := CreateIterable( //nolint:wrapcheck
66-
func(*Event, error) (*Event, error) {
67-
if resp.EventID == nil {
68-
return nil, reportError("received unexpected response from the push endpoint, eventID must not be undefined")
69-
}
70-
71-
return c.GetEvent(c.NewApiGetEventRequest(resp.RunID, *resp.EventID))
72-
},
73-
func(response *Event, err error) (bool, error) {
74-
var apiErr *APIError
75-
if errors.As(err, &apiErr) {
76-
return apiErr.Status != 404, nil
77-
}
78-
79-
return true, err
80-
},
81-
WithTimeout(func(count int) time.Duration { return time.Duration(min(500*count, 5000)) * time.Millisecond }), WithMaxRetries(50),
82-
)
83-
if err != nil {
84-
return nil, err
68+
69+
if conf.waitForTasks && len(responses) > 0 && (len(responses)%waitBatchSize == 0 || i == len(objects)-1) {
70+
var waitableResponses []WatchResponse
71+
72+
if len(responses) > offset+waitBatchSize {
73+
waitableResponses = responses[offset:waitBatchSize]
74+
} else {
75+
waitableResponses = responses[offset:]
8576
}
77+
78+
for _, resp := range waitableResponses {
79+
_, err := CreateIterable(
80+
func(*Event, error) (*Event, error) {
81+
if resp.EventID == nil {
82+
return nil, reportError("received unexpected response from the push endpoint, eventID must not be undefined")
83+
}
84+
85+
return c.GetEvent(c.NewApiGetEventRequest(resp.RunID, *resp.EventID))
86+
},
87+
func(response *Event, err error) (bool, error) {
88+
var apiErr *APIError
89+
if errors.As(err, &apiErr) {
90+
return apiErr.Status != 404, nil
91+
}
92+
93+
return true, err
94+
},
95+
WithTimeout(func(count int) time.Duration { return time.Duration(min(500*count, 5000)) * time.Millisecond }), WithMaxRetries(50),
96+
)
97+
if err != nil {
98+
return nil, err
99+
}
100+
}
101+
102+
offset += waitBatchSize
86103
}
87104
}
88105

‎templates/java/api.mustache

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import java.util.function.IntUnaryOperator;
2020

2121
import java.time.Duration;
2222
import java.util.EnumSet;
23+
import java.util.Iterator;
2324
import java.util.Random;
2425
import java.util.Collections;
2526
import java.util.ArrayList;

‎templates/java/api_helpers.mustache

Lines changed: 48 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -44,58 +44,62 @@ public <T> List<WatchResponse> chunkedPush(
4444
) {
4545
List<WatchResponse> responses = new ArrayList<>();
4646
List<T> records = new ArrayList<>();
47+
int offset = 0;
48+
int waitBatchSize = batchSize / 10;
49+
if (waitBatchSize < 1) {
50+
waitBatchSize = batchSize;
51+
}
4752

48-
for (T item : objects) {
49-
if (records.size() == batchSize) {
50-
WatchResponse watch =
51-
this.push(
52-
indexName,
53-
new PushTaskPayload().setAction(action).setRecords(this.objectsToPushTaskRecords(records)),
54-
waitForTasks,
55-
referenceIndexName,
56-
requestOptions
57-
);
53+
Iterator<T> it = objects.iterator();
54+
T current = it.next();
55+
56+
while (true) {
57+
records.add(current);
58+
59+
if (records.size() == batchSize || !it.hasNext()) {
60+
WatchResponse watch = this.push(
61+
indexName,
62+
new PushTaskPayload().setAction(action).setRecords(this.objectsToPushTaskRecords(records)),
63+
waitForTasks,
64+
referenceIndexName,
65+
requestOptions
66+
);
5867
responses.add(watch);
5968
records.clear();
6069
}
6170

62-
records.add(item);
63-
}
71+
if (waitForTasks && responses.size() > 0 && (responses.size() % waitBatchSize == 0 || !it.hasNext())) {
72+
responses
73+
.subList(offset, Math.min(offset + waitBatchSize, responses.size()))
74+
.forEach(response -> {
75+
TaskUtils.retryUntil(
76+
() -> {
77+
try {
78+
return this.getEvent(response.getRunID(), response.getEventID());
79+
} catch (AlgoliaApiException e) {
80+
if (e.getStatusCode() == 404) {
81+
return null;
82+
}
83+
84+
throw e;
85+
}
86+
},
87+
(Event resp) -> {
88+
return resp != null;
89+
},
90+
50,
91+
null
92+
);
93+
});
6494

65-
if (records.size() > 0) {
66-
WatchResponse watch =
67-
this.push(
68-
indexName,
69-
new PushTaskPayload().setAction(action).setRecords(this.objectsToPushTaskRecords(records)),
70-
waitForTasks,
71-
referenceIndexName,
72-
requestOptions
73-
);
74-
responses.add(watch);
75-
}
95+
offset += waitBatchSize;
96+
}
7697

77-
if (waitForTasks) {
78-
responses.forEach(response -> {
79-
TaskUtils.retryUntil(
80-
() -> {
81-
try {
82-
return this.getEvent(response.getRunID(), response.getEventID());
83-
} catch (AlgoliaApiException e) {
84-
if (e.getStatusCode() == 404) {
85-
return null;
86-
}
98+
if (!it.hasNext()) {
99+
break;
100+
}
87101

88-
throw e;
89-
}
90-
},
91-
(Event resp) -> {
92-
return resp != null;
93-
},
94-
50,
95-
null
96-
);
97-
}
98-
);
102+
current = it.next();
99103
}
100104

101105
return responses;

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy