Data management API · Apache Druid

This topic describes the data management API endpoints for Apache Druid. This includes information on how to mark segments as used or unused and delete them from Druid.

Note that while you can mark segments as used by sending POST requests to the datasource, the Coordinator may subsequently mark segments as unused if they meet any configured Drop rules. Even if these API requests update segments to used, you still need to configure a Load rule to load them onto Historical processes.

When you use these APIs concurrently with an indexing task or a kill task, the behavior is undefined. Some segments may be terminated, while others are marked as used. Furthermore, it is possible that all segments could be unused, yet an indexing task might still be able to read data from these segments and complete successfully.

In this topic, http://ROUTER_IP:ROUTER_PORT is a placeholder for your Router service address and port. Replace it with the information for your deployment. For example, use http://localhost:8888 for quickstart deployments.

Avoid using indexing or kill tasks and these APIs at the same time for the same datasource and time chunk.

Segment management

Update a group of segments as `unused`

Updates the state a group of segments as unused using the an array of segment IDs or an interval. Pass the array of segment IDs or interval as a JSON object in the request body.

Interval specifies the start and end times as IS0 8601 strings. interval=(start/end) where start is inclusive and end is non-inclusive. Only the segments completely contained within the specified interval will be updated, partially overlapping segments will not be affected.

URL

POST /druid/coordinator/v1/datasources/:dataSource/markUnused

Request body

The group of segments is sent as a JSON request payload with the following properties:

Property	Description	Example
`interval`	The interval of segments.	`"2015-09-12T03:00:00.000Z/2015-09-12T05:00:00.000Z"`
`segmentIds`	Array of segment IDs.	`["segmentId1", "segmentId2"]`

Responses

200 SUCCESS

204 NO CONTENT

400 BAD REQUEST

Successfully updated segments

Invalid dataSource name

Invalid request payload

Sample request

The following example updates two segments with IDs wikipedia_hour_2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z_2023-08-10T04:12:03.860Z and wikipedia_hour_2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z_2023-08-10T04:12:03.860Z as unused.

cURL

HTTP

curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour/markUnused" \
--header 'Content-Type: application/json' \
--data '{
    "segmentIds": [
        "wikipedia_hour_2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z_2023-08-10T04:12:03.860Z",
        "wikipedia_hour_2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z_2023-08-10T04:12:03.860Z"
    ]
}'

POST /druid/coordinator/v1/datasources/wikipedia_hour/markUnused HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Content-Length: 230

{
    "segmentIds": [
        "wikipedia_hour_2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z_2023-08-10T04:12:03.860Z",
        "wikipedia_hour_2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z_2023-08-10T04:12:03.860Z"
    ]
}

Sample response

Click to show sample response

{
    "numChangedSegments": 2
}

Update all datasource segments as `unused`

Updates the state of all segments of a datasource to unused. This is a "soft delete" of the segments from Historicals.

Note that this endpoint returns an HTTP 200 OK message code even if the datasource name does not exist.

URL

DELETE /druid/coordinator/v1/datasources/:dataSource

Responses

200 SUCCESS

Successfully updated segments

Sample request

cURL

HTTP

curl --request DELETE "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour"

DELETE /druid/coordinator/v1/datasources/wikipedia_hour HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT

Sample response

Click to show sample response

{
    "numChangedSegments": 24
}

Update a segment as `unused`

Updates the state a segment as unused using the segment's ID. This is a "soft delete" of the segment from Historicals. To undo this delete, mark the segments as used.

Note that this endpoint returns an HTTP 200 OK message code even if the segment ID or datasource name does not exist.

URL

DELETE /druid/coordinator/v1/datasources/:dataSource/segments/:segmentId

Header

The following header is required to make a call to this endpoint.

Content-Type: application/json
Accept: application/json, text/plain

Responses

200 SUCCESS

Successfully updated segment

Sample request

The following example updates the segment wikipedia_hour_2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z_2023-08-10T04:12:03.860Z from datasource wikipedia_hour as unused.

cURL

HTTP

curl --request DELETE "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z_2023-08-10T04:12:03.860Z" \
--header 'Content-Type: application/json' \
--header 'Accept: application/json, text/plain'

DELETE /druid/coordinator/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T16:00:00.000Z_2015-09-12T17:00:00.000Z_2023-08-10T04:12:03.860Z HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Accept: application/json, text/plain

Sample response

Click to show sample response

{
    "segmentStateChanged": true
}

Update a group of segments as `used`

Updates the state a group of segments as used using the an array of segment IDs or an interval. Pass the array of segment IDs or interval as a JSON object in the request body.

Interval specifies the start and end times as ISO 8601 strings. interval=(start/end) where start is inclusive and end is non-inclusive. Only the segments completely contained within the specified interval will be updated, partially overlapping segments will not be affected.

URL

POST /druid/coordinator/v1/datasources/:dataSource/markUsed

Request body

The group of segments is sent as a JSON request payload with the following properties:

Property	Type	Description	Example
`interval`	ISO-8601	The interval of segments.	`"2015-09-12T03:00:00.000Z/2015-09-12T05:00:00.000Z"`
`segmentIds`	String	Array of segment IDs.	`["segmentId1", "segmentId2"]`

Responses

200 SUCCESS

204 NO CONTENT

400 BAD REQUEST

Successfully updated segments

Invalid dataSource name

Invalid request payload

Sample request

cURL

HTTP

curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour/markUsed" \
--header 'Content-Type: application/json' \
--data '{
    "segmentIds": [
        "wikipedia_hour_2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z_2023-08-10T04:12:03.860Z",
        "wikipedia_hour_2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z_2023-08-10T04:12:03.860Z"
    ]
}'

POST /druid/coordinator/v1/datasources/wikipedia_hour/markUsed HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Content-Length: 230

{
    "segmentIds": [
        "wikipedia_hour_2015-09-12T14:00:00.000Z_2015-09-12T15:00:00.000Z_2023-08-10T04:12:03.860Z",
        "wikipedia_hour_2015-09-12T04:00:00.000Z_2015-09-12T05:00:00.000Z_2023-08-10T04:12:03.860Z"
    ]
}

Sample response

Click to show sample response

{
    "numChangedSegments": 2
}

Update all datasource segments as `used`

Updates the state of all unused segments of a datasource to used. If there are no segments eligible to be marked as used, this endpoint will return the property "numChangedSegments" and the value 0.

Note that this endpoint returns an HTTP 200 OK message code even if the datasource name does not exist.

URL

POST /druid/coordinator/v1/datasources/:dataSource

Header

The following header is required to make a call to this endpoint.

Content-Type: application/json
Accept: application/json, text/plain

Responses

200 SUCCESS

Successfully updated segments

Sample request

The following example updates all unused segments of wikipedia_hour to used. wikipedia_hour contains one unused segment eligible to be marked as used.

cURL

HTTP

curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour" \
--header 'Content-Type: application/json' \
--header 'Accept: application/json, text/plain'

POST /druid/coordinator/v1/datasources/wikipedia_hour HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Accept: application/json, text/plain

Sample response

Click to show sample response

{
    "numChangedSegments": 1
}

Update a segment as `used`

Updates the state a segment as used using the segment's ID.

URL

POST /druid/coordinator/v1/datasources/segments/:segmentId

Header

The following header is required to make a call to this endpoint.

Content-Type: application/json
Accept: application/json, text/plain

Responses

200 SUCCESS

Successfully updated segments

Sample request

The following example updates the segment with ID wikipedia_hour_2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z_2023-08-10T04:12:03.860Z to used.

cURL

HTTP

curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z_2023-08-10T04:12:03.860Z" \
--header 'Content-Type: application/json' \
--header 'Accept: application/json, text/plain'

POST /druid/coordinator/v1/datasources/wikipedia_hour/segments/wikipedia_hour_2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z_2023-08-10T04:12:03.860Z HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Accept: application/json, text/plain

Sample response

Click to show sample response

{
    "segmentStateChanged": true
}

Segment deletion

Permanently delete segments with a kill task

Sends a Kill task for a given interval and datasource. The interval value is an ISO-8601 string delimited by _. This endpoint permanently deletes all metadata about unused segments and removes them from deep storage.

Note that this endpoint returns an HTTP 200 OK message code even if the datasource name does not exist.

This succeeds the deprecated endpoint: DELETE /druid/coordinator/v1/datasources/:dataSource?kill=true&interval=:interval

URL

DELETE /druid/coordinator/v1/datasources/:dataSource/intervals/:interval

Responses

200 SUCCESS

Successfully sent kill task

Sample request

The following example sends a kill task to permanently delete segments in the datasource wikipedia_hour from the interval 2015-09-12 to 2015-09-13.

cURL

HTTP

curl --request DELETE "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/datasources/wikipedia_hour/intervals/2015-09-12_2015-09-13"

DELETE /druid/coordinator/v1/datasources/wikipedia_hour/intervals/2015-09-12_2015-09-13 HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT

Sample response

A successful request returns an HTTP 200 OK and an empty response body.

Segment management

Update a group of segments as unused

URL

Request body

Responses

Sample request

Sample response

Update all datasource segments as unused

URL

Responses

Sample request

Sample response

Update a segment as unused

URL

Header

Responses

Sample request

Sample response

Update a group of segments as used

URL

Request body

Responses

Sample request

Sample response

Update all datasource segments as used

URL

Header

Responses

Sample request

Sample response

Update a segment as used

URL

Header

Responses

Sample request

Sample response

Segment deletion

Permanently delete segments with a kill task

URL

Responses

Sample request

Sample response

Update a group of segments as `unused`

Update all datasource segments as `unused`

Update a segment as `unused`

Update a group of segments as `used`

Update all datasource segments as `used`

Update a segment as `used`