LogoLogo
Why Unitary?
  • Get started
    • 馃憢Step-by-step instructions
  • API References
    • 鈩癸笍Features and requirements
    • 鈻讹笍API Authentication
    • 鈽戯笍Standard & Premium: Policy Classification endpoints
      • 馃攦Add-on: synchronous endpoints
    • 馃泜Virtual Moderator endpoint
    • #锔忊儯Items and Characteristics endpoints
      • *锔忊儯List of Item and Characteristics
    • 鈴猴笍Detoxify endpoints
    • 鈫旓笍How to select thresholds
    • 鈫笍Integrating Webhooks
Powered by GitBook
On this page

Was this helpful?

  1. API References

Items and Characteristics endpoints

PreviousVirtual Moderator endpointNextList of Item and Characteristics

Last updated 1 year ago

Was this helpful?

The Items & Characteristics endpoints surface the items and characteristics detected in an image or video, including sound and speech. Each item or characteristic has a prevalence score which represents the extent to which it is present in a given piece of content. Unitary can also analyse the text that is shared alongside the image or video.

You can use these scores in mappings or models to implement safety policies, for example setting a numerical threshold for automated flagging of content or in a linear regression model (read more in the ).

Unitary output's over 50 different classes you can find onList of Item and Characteristics

This guide assumes to have your API key already setup. If this is not the case, please email at

1. Authenticate

Please follow the API Authentication instructions first in order to authenticate with the API. This will generate a token that is valid for 24 hours and must be used in subsequent API requests.

2. Sending your first request

Depending on your use case, pick an image or a video and then use one of the following POST endpoints to send your first request.

POST endpoint reference

Example code

Content supplied as a binary file upload:

curl --location --request POST "https://api.unitary.ai/v1/classify/items-characteristics/image" \
--header "Authorization: Bearer {API_TOKEN}" \
--header "Content-Type: multipart/form-data" \
--form "file=@{FILE};type=image/{FILE_EXTENSION}" \
--form "caption={CAPTION}"

Content supplied as a resource URL:

curl --location --request POST "https://api.unitary.ai/v1/classify/items-characteristics/image" \
--header "Authorization: Bearer {API_TOKEN}" \
--form "caption={CAPTION}" \
--form "url={RESOURCE_URL}"
import requests

api_token = "your_api_token_here"
file_path = "/path/to/your/file.jpg"
caption = "your_caption_here"
url = "https://api.unitary.ai/v1/classify/items-characteristics/image"

# For sending a file
files = {'file': (file_path, open(file_path, 'rb'), 'image/jpeg')}
data = {'caption': caption}
headers = {'Authorization': f'Bearer {api_token}'}

response = requests.post(url, files=files, data=data, headers=headers)
job_id = resonse.text["job_id"]

print(job_id)

# For sending a URL
data = {
    'caption': caption,
    'url': 'your_resource_url_here'
}
headers = {'Authorization': f'Bearer {api_token}'}

response = requests.post(url, data=data, headers=headers)
job_id = resonse.text["job_id"]

print(job_id)
const fetch = require('node-fetch');
const FormData = require('form-data');
const fs = require('fs');

const apiToken = 'your_api_token_here';
const file_path = '/path/to/your/file.jpg';
const caption = 'your_caption_here';
const url = 'https://api.unitary.ai/v1/classify/items-characteristics/image';

// For sending a file
const formDataFile = new FormData();
formDataFile.append('file', fs.createReadStream(file_path), { type: 'image/jpeg' });
formDataFile.append('caption', caption);

fetch(url, {
  method: 'POST',
  body: formDataFile,
  headers: {
    'Authorization': `Bearer ${apiToken}`,
  },
})
.then(response => response.text())
.then(result => console.log(result["job_id"]))
.catch(error => console.log('error', error));

// For sending a URL
const dataUrl = new URLSearchParams();
dataUrl.append('caption', caption);
dataUrl.append('url', 'your_resource_url_here');

fetch(url, {
  method: 'POST',
  body: dataUrl,
  headers: {
    'Authorization': `Bearer ${apiToken}`,
    'Content-Type': 'application/x-www-form-urlencoded',
  },
})
.then(response => response.text())
.then(result => console.log(result["job_id"]))
.catch(error => console.log('error', error));

3. Get results back via the GET endpoint

Using the GET endpoint may be inadequate for large scale! The recommendation is using the "callback_url" parameter instead. Please check the Integrating Webhooks guide for more information.

GET endpoint reference

Example code

curl --location --request GET "https://api.unitary.ai/v1/classify/items-characteristics/image/{JOB_ID}" \
--header "Authorization: Bearer {BEARER_TOKEN}"
import requests

BEARER_TOKEN = "{BEARER_TOKEN}"
JOB_ID = "{JOB_ID}"
url = f"https://api.unitary.ai/v1/classify/items-characteristics/image/{JOB_ID}"

payload={}
headers = {
  'Authorization': f'Bearer {BEARER_TOKEN}'
}

response = requests.request("GET", url, headers=headers, data=payload)

results = response["results"]
violence = results["violence"]
has_firearms = violence["firearm"] > 0.8

print(has_firearms)
const fetch = require("node-fetch"); 
const BEARER_TOKEN = "{BEARER_TOKEN}";
const JOB_ID = "{JOB_ID}";
const url = `https://api.unitary.ai/v1/classify/items-characteristics/image/${JOB_ID}`;

fetch(url, {
  method: "GET",
  headers: {
    'Authorization': `Bearer ${BEARER_TOKEN}`,
  },
})
.then(response => response.json())
.then(data => {
  const results = data["results"];
  const violence = results["violence"];
  const has_firearms = violence["firearm"] > 0.8;
  console.log(has_firearms);
})
.catch((error) => {
  console.error('Error:', error);
});

Depending on the processing status of the job at the time of request, you can get any of the following as the response from the GET endpoints:

{
  "status": "done",
  "results": {
    "violence": {
      "violence": 0.006823897361755371,
      "firearm": 0.06859919428825378,
      "knife": 0.03536936640739441,
      "violent_knife": 0.03841346502304077
    },
    "substances": {
      "alcohol": 0.09190595149993896,
      "drink": 0.11272519826889038,
      "smoking_and_tobacco": 0.1175956130027771,
      "marijuana": 0.10564917325973511,
      "pills": 0.1289033591747284,
      "recreational_pills": 0.12389451265335083
    },
    "nsfw": {
      "adult_content": 0.7132323980331421,
      "suggestive": 0.9982926249504089,
      "adult_toys": 2.8699636459350586e-05,
      "medical": 0.9637787342071533,
      "over_18": 0.8957071900367737,
      "exposed_anus": 0.0003555417060852051,
      "exposed_armpits": 0.1219978928565979,
      "exposed_belly": 0.2361663281917572,
      "covered_belly": 0.004645615816116333,
      "covered_buttocks": 0.007969379425048828,
      "exposed_buttocks": 0.040920644998550415,
      "covered_feet": 0.004700213670730591,
      "exposed_feet": 0.0017207860946655273,
      "covered_breast_f": 0.5827286839485168,
      "exposed_breast_f": 0.5190947651863098,
      "covered_genitalia_f": 0.11009037494659424,
      "exposed_genitalia_f": 0.003075540065765381,
      "exposed_breast_m": 0.0006746351718902588,
      "exposed_genitalia_m": 0.005911082029342651
    },
    "hate_symbols": {
      "confederate_flag": 0.10465598106384277,
      "pepe_frog": 0.009708642959594727,
      "nazi_swastika": 0.07055380940437317
    },
    "caption_language_toxicity": {
      "toxicity": 0.0003523826599121094,
      "severe_toxicity": 4.458427429199219e-05,
      "obscene": 0.00036585330963134766,
      "insult": 0.0004227757453918457,
      "identity_attack": 0.00011718273162841797,
      "threat": 5.6624412536621094e-05,
      "sexual_explicit": 4.363059997558594e-05
    },
    "visual_content": {
      "artistic": 0.00594728346914053,
      "comic": 0.0004559260851237923,
      "meme": 0.00048818273353390396,
      "screenshot": 0.009867854416370392,
      "map": 2.6955993234878406e-05,
      "poster_cover": 4.053880275023403e-06,
      "game_screenshot": 4.549684308585711e-05,
      "face_filter": 0.008053907193243504,
      "promo_info_graphic": 6.63170067127794e-06,
      "photo": 0.9751037955284119
    },
    "other": {
      "child": 0.05087965726852417,
      "middle_finger_gesture": 0.0770421028137207,
      "toy": 0.09503969550132751,
      "gambling_machine": 0.09784704446792603,
      "face_f": 0.501373291015625,
      "face_m": 0.0010322332382202148
    },
    "url": "{RESOURCE_URL}"
  },
  "msg": null
}
{
  "status": "queued"
}
{
  "status": "failed",
  "msg": "the error message for the request"
}

4. Thresholding

Please follow our thresholding guide: How to select thresholds

5. Webhooks

(optional but encouraged)

In order to have a scalable end-to-end integration, the last step is to set up the receiving of webhooks as described in the following guide: Integrating Webhooks

#锔忊儯
guide for selecting thresholds
support@unitary.ai

Retrieve the results of a queued Items & Characteristics classification

get

This endpoint returns the results of a previously queued classify job.

The response for each item is a prevalence score between 0 and 1 that represents how present it is in the content. 1 is highest prevalence and 0 is lowest prevalence.

You can use these scores in mappings or models to implement safety policies, for example setting a numerical threshold for automated flagging of content or in a linear regression model.

Note that the score meaning and distribution is different for each item: for example, a 0.7 for pepe won't translate to the same likelihood of harmful content as a 0.7 for confederate flag. Therefore you should calculate an individual threshold for each item. For best performance, we suggest calculating a threshold using labelled data, optimising for the balance of precision and recall best suited to your needs. The Unitary team is happy to provide guidance on this process.

Authorizations
Path parameters
job_idstring 路 uuid4Required
Responses
200
Successful Response
application/json
404
Not found
422
Validation Error
application/json
get
GET /classify/items-characteristics/image/{job_id} HTTP/1.1
Host: 
Authorization: Bearer JWT
Accept: */*
{
  "status": "done",
  "results": {
    "violence": {
      "violence": 0.54324,
      "firearm": 0.54324,
      "knife": 0.54324,
      "violent_knife": 0.54324
    },
    "substances": {
      "alcohol": 0.54324,
      "drink": 0.54324,
      "smoking_and_tobacco": 0.54324,
      "marijuana": 0.54324,
      "pills": 0.54324,
      "recreational_pills": 0.54324
    },
    "nsfw": {
      "adult_content": 0.54324,
      "suggestive": 0.54324,
      "adult_toys": 0.54324,
      "medical": 0.54324,
      "over_18": 0.54324,
      "exposed_anus": 0.54324,
      "exposed_armpits": 0.54324,
      "exposed_belly": 0.54324,
      "covered_belly": 0.54324,
      "covered_buttocks": 0.54324,
      "exposed_buttocks": 0.54324,
      "covered_feet": 0.54324,
      "exposed_feet": 0.54324,
      "covered_breast_f": 0.54324,
      "exposed_breast_f": 0.54324,
      "covered_genitalia_f": 0.54324,
      "exposed_genitalia_f": 0.54324,
      "exposed_breast_m": 0.54324,
      "exposed_genitalia_m": 0.54324
    },
    "hate_symbols": {
      "confederate_flag": 0.54324,
      "pepe_frog": 0.54324,
      "nazi_swastika": 0.54324
    },
    "ocr_language_toxicity": {
      "toxicity": 0.54324,
      "severe_toxicity": 0.54324,
      "obscene": 0.54324,
      "insult": 0.54324,
      "identity_attack": 0.54324,
      "threat": 0.54324,
      "sexual_explicit": 0.54324
    },
    "audio_language_toxicity": {
      "toxicity": 0.54324,
      "severe_toxicity": 0.54324,
      "obscene": 0.54324,
      "insult": 0.54324,
      "identity_attack": 0.54324,
      "threat": 0.54324,
      "sexual_explicit": 0.54324
    },
    "caption_language_toxicity": {
      "toxicity": 0.54324,
      "severe_toxicity": 0.54324,
      "obscene": 0.54324,
      "insult": 0.54324,
      "identity_attack": 0.54324,
      "threat": 0.54324,
      "sexual_explicit": 0.54324
    },
    "visual_content": {
      "artistic": 0.54324,
      "comic": 0.54324,
      "meme": 0.54324,
      "screenshot": 0.54324,
      "map": 0.54324,
      "poster_cover": 0.54324,
      "game_screenshot": 0.54324,
      "face_filter": 0.54324,
      "promo_info_graphic": 0.54324,
      "photo": 0.54324
    },
    "other": {
      "child": 0.54324,
      "middle_finger_gesture": 0.54324,
      "toy": 0.54324,
      "gambling_machine": 0.54324,
      "face_f": 0.54324,
      "face_m": 0.54324
    },
    "ocr_texts": [
      "text"
    ],
    "audio_texts": [
      "text"
    ],
    "url": "https://example.com"
  },
  "msg": "text"
}

Retrieve the results of a queued Items & Characteristics classification

get

This endpoint returns the results of a previously queued classify job.

The response for each item is a prevalence score between 0 and 1 that represents how present it is in the content. 1 is highest prevalence and 0 is lowest prevalence.

You can use these scores in mappings or models to implement safety policies, for example setting a numerical threshold for automated flagging of content or in a linear regression model.

Note that the score meaning and distribution is different for each item: for example, a 0.7 for pepe won't translate to the same likelihood of harmful content as a 0.7 for confederate flag. Therefore you should calculate an individual threshold for each item. For best performance, we suggest calculating a threshold using labelled data, optimising for the balance of precision and recall best suited to your needs. The Unitary team is happy to provide guidance on this process.

Authorizations
Path parameters
job_idstring 路 uuid4Required
Responses
200
Successful Response
application/json
404
Not found
422
Validation Error
application/json
get
GET /classify/items-characteristics/video/{job_id} HTTP/1.1
Host: 
Authorization: Bearer JWT
Accept: */*
{
  "status": "done",
  "results": {
    "violence": {
      "violence": 0.54324,
      "firearm": 0.54324,
      "knife": 0.54324,
      "violent_knife": 0.54324
    },
    "substances": {
      "alcohol": 0.54324,
      "drink": 0.54324,
      "smoking_and_tobacco": 0.54324,
      "marijuana": 0.54324,
      "pills": 0.54324,
      "recreational_pills": 0.54324
    },
    "nsfw": {
      "adult_content": 0.54324,
      "suggestive": 0.54324,
      "adult_toys": 0.54324,
      "medical": 0.54324,
      "over_18": 0.54324,
      "exposed_anus": 0.54324,
      "exposed_armpits": 0.54324,
      "exposed_belly": 0.54324,
      "covered_belly": 0.54324,
      "covered_buttocks": 0.54324,
      "exposed_buttocks": 0.54324,
      "covered_feet": 0.54324,
      "exposed_feet": 0.54324,
      "covered_breast_f": 0.54324,
      "exposed_breast_f": 0.54324,
      "covered_genitalia_f": 0.54324,
      "exposed_genitalia_f": 0.54324,
      "exposed_breast_m": 0.54324,
      "exposed_genitalia_m": 0.54324
    },
    "hate_symbols": {
      "confederate_flag": 0.54324,
      "pepe_frog": 0.54324,
      "nazi_swastika": 0.54324
    },
    "ocr_language_toxicity": {
      "toxicity": 0.54324,
      "severe_toxicity": 0.54324,
      "obscene": 0.54324,
      "insult": 0.54324,
      "identity_attack": 0.54324,
      "threat": 0.54324,
      "sexual_explicit": 0.54324
    },
    "audio_language_toxicity": {
      "toxicity": 0.54324,
      "severe_toxicity": 0.54324,
      "obscene": 0.54324,
      "insult": 0.54324,
      "identity_attack": 0.54324,
      "threat": 0.54324,
      "sexual_explicit": 0.54324
    },
    "caption_language_toxicity": {
      "toxicity": 0.54324,
      "severe_toxicity": 0.54324,
      "obscene": 0.54324,
      "insult": 0.54324,
      "identity_attack": 0.54324,
      "threat": 0.54324,
      "sexual_explicit": 0.54324
    },
    "visual_content": {
      "artistic": 0.54324,
      "comic": 0.54324,
      "meme": 0.54324,
      "screenshot": 0.54324,
      "map": 0.54324,
      "poster_cover": 0.54324,
      "game_screenshot": 0.54324,
      "face_filter": 0.54324,
      "promo_info_graphic": 0.54324,
      "photo": 0.54324
    },
    "other": {
      "child": 0.54324,
      "middle_finger_gesture": 0.54324,
      "toy": 0.54324,
      "gambling_machine": 0.54324,
      "face_f": 0.54324,
      "face_m": 0.54324
    },
    "ocr_texts": [
      "text"
    ],
    "audio_texts": [
      "text"
    ],
    "url": "https://example.com",
    "metadata": {
      "width": 1,
      "height": 1,
      "fps": 1,
      "duration": 1,
      "seconds_processed": 1
    }
  },
  "msg": "text"
}
  • 1. Authenticate
  • 2. Sending your first request
  • POST endpoint reference
  • POSTQueue an image for Items & Characteristics classification
  • POSTQueue a video for Items & Characteristics classification
  • Example code
  • 3. Get results back via the GET endpoint
  • GET endpoint reference
  • GETRetrieve the results of a queued Items & Characteristics classification
  • GETRetrieve the results of a queued Items & Characteristics classification
  • Example code
  • 4. Thresholding
  • 5. Webhooks

Queue an image for Items & Characteristics classification

post

This endpoint queues a job that predicts the Items & Characteristics categories of a single image.

The request must contain either a file (a multi-part form encoded image file) or a url form field (a string containing a URL to download the image file from).

Example CURL:

curl --location --request POST https://api.unitary.ai/v1/classify/items-characteristics/image --header "Authorization: Bearer {API_TOKEN}" --header "Content-Type: multipart/form-data" --form "url={RESOURCE_URL}"

A job ID will be returned in the response.

If a callback_url form field is included in the request then the results webhook will be sent as a POST request to this URL once the image has been classified.

Alternatively results can be obtained by making a GET request to the results endpoint using the job ID.

Webhooks should be implemented for any use of this endpoint at scale to prevent unnecessary polling of the results endpoint.

Authorizations
Body
callback_urlstring 路 uri 路 min: 1 路 max: 2083Optional
captionstringOptional
filestring 路 binaryOptional
urlstring 路 uri 路 min: 1 路 max: 65536Optional
Responses
200
Successful Response
application/json
404
Not found
422
Validation Error
application/json
post
POST /classify/items-characteristics/image HTTP/1.1
Host: 
Authorization: Bearer JWT
Content-Type: multipart/form-data
Accept: */*
Content-Length: 99

{
  "callback_url": "https://example.com",
  "caption": "text",
  "file": "binary",
  "url": "https://example.com"
}
{
  "job_id": "text"
}

Queue a video for Items & Characteristics classification

post

This endpoint queues a job that predicts the Items & Characteristics categories of a single video.

The request must contain either a file (a multi-part form encoded video file) or a url form field (a string containing a URL to download the video file from).

Example CURL:

curl --location --request POST https://api.unitary.ai/v1/classify/items-characteristics/video --header "Authorization: Bearer {API_TOKEN}" --header "Content-Type: multipart/form-data" --form "url={RESOURCE_URL}"

A job ID will be returned in the response.

If a callback_url form field is included in the request then the results webhook will be sent as a POST request to this URL once the image has been classified.

Alternatively results can be obtained by making a GET request to the results endpoint using the job ID.

Webhooks should be implemented for any use of this endpoint at scale to prevent unnecessary polling of the results endpoint.

Authorizations
Body
callback_urlstring 路 uri 路 min: 1 路 max: 2083Optional
captionstringOptional
filestring 路 binaryOptional
urlstring 路 uri 路 min: 1 路 max: 65536Optional
Responses
200
Successful Response
application/json
404
Not found
422
Validation Error
application/json
post
POST /classify/items-characteristics/video HTTP/1.1
Host: 
Authorization: Bearer JWT
Content-Type: multipart/form-data
Accept: */*
Content-Length: 99

{
  "callback_url": "https://example.com",
  "caption": "text",
  "file": "binary",
  "url": "https://example.com"
}
{
  "job_id": "text"
}