Object Detection
This module allows the detection of three diferent types of objects (people, vehicles and two wheeler vehicles) using a SDK called DeepStream running on a service in a Jetson Nano.
The main functions of the program are:
- tiler_src_pad_buffer_probe(pad,info,u_data) - tiler_sink_pad_buffer_probe will extract metadata received on OSD sink pad and update params for drawing rectangle, object information etc.
- create_source_bin(index,uri) - Create a source GstBin to abstract this bin’s content from the rest of the pipeline
- cb_newpad(decodebin, decoder_src_pad,data) - Gets the source bin ghost pad and checks if the pad created by the decodebin is for video and not audio
- main() - Creates a Pipeline element that will form a connection of other elements, a nvstreammux instance to form batches from one or more sources, a tiler and an egl sink
def tiler_src_pad_buffer_probe(pad,info,u_data):
frame_number=0
num_rects=0
is_first_object=True
gst_buffer = info.get_buffer()
if not gst_buffer:
print("Unable to get GstBuffer ")
return
# Retrieve batch metadata from the gst_buffer
# Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
# C address of gst_buffer as input, which is obtained with hash(gst_buffer)
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
try:
# Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
# The casting is done by pyds.NvDsFrameMeta.cast()
# The casting also keeps ownership of the underlying memory
# in the C code, so the Python garbage collector will leave
# it alone.
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
except StopIteration:
continue
is_first_object = True
frame_number=frame_meta.frame_num
l_obj=frame_meta.obj_meta_list
num_rects = frame_meta.num_obj_meta
obj_counter = {
PGIE_CLASS_ID_VEHICLE:0,
PGIE_CLASS_ID_PERSON:0,
PGIE_CLASS_ID_BICYCLE:0,
PGIE_CLASS_ID_ROADSIGN:0
}
while l_obj is not None:
try:
# Casting l_obj.data to pyds.NvDsObjectMeta
obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
except StopIteration:
continue
obj_counter[obj_meta.class_id] += 1
# Ideally NVDS_EVENT_MSG_META should be attached to buffer by the
# component implementing detection / recognition logic.
# Here it demonstrates how to use / attach that meta data.
if(is_first_object):
# Allocating an NvDsEventMsgMeta instance and getting reference
# to it. The underlying memory is not manged by Python so that
# downstream plugins can access it. Otherwise the garbage collector
# will free it when this probe exits.
msg_meta=pyds.alloc_nvds_event_msg_meta()
msg_meta.bbox.top = obj_meta.rect_params.top #Holds top coordinate of the box in pixels
msg_meta.bbox.left = obj_meta.rect_params.left #Holds left coordinate of the box in pixels
msg_meta.bbox.width = obj_meta.rect_params.width #Holds width of the box in pixels.
msg_meta.bbox.height = obj_meta.rect_params.height #Holds height of the box in pixels
if msg_meta.bbox.left < 300: #retira area da Rua da Pega
obj_counter[obj_meta.class_id] -= 1
is_first_object = True
try:
l_obj=l_obj.next
except StopIteration:
break
global num_people
global num_vehicles
global num_twowheelers
num_people = obj_counter[PGIE_CLASS_ID_PERSON]
num_vehicles = obj_counter[PGIE_CLASS_ID_VEHICLE]
num_twowheelers = obj_counter[PGIE_CLASS_ID_BICYCLE]
# Get frame rate through this probe
fps_streams["stream{0}".format(frame_meta.pad_index)].get_fps()
try:
l_frame=l_frame.next
except StopIteration:
break
return Gst.PadProbeReturn.OK
There are 2 types of values: unique values and current values.
The current values are captured every second after analysing 12.4 video frames. This means that every second a message with the number of detected people, vehicles and two wheeler vehicles is sent to the local broker.
The unique values are sent in a time interval of 60 seconds and they are the sum of every new object that appeared during that time interval in each category. That means that if during a minute 22 new people apear on the video feed, the people unique value at the end of that minute will be 22.
This captured data is sent to a local broker in 6 different topics:
#current
currentppl_topic = "detection/people/current"
currentvehicles_topic = "detection/vehicle/current"
currenttwowheelers_topic = "detection/twowheelers/current"
#unique
uniqueppl_topic = "detection/people/unique"
uniquevehicles_topic = "detection/vehicle/unique"
uniquetwowheelers_topic = "detection/twowheelers/unique"
Since the data has to be sent in different time intervals, its handling has to be separate.
The real time data relating to the number of objects detected is sent every second. To prevent sending several messages with the value 0 for hours (like for example during the night) to the broker, the code was changed in order to supress sending repeated zeros. To do this, every time a message is sent, it has to be checked if the previous message was a 0 and if the current message is also a 0.
The median of people that pass in a certain area during an exact hour is obtained by calculating the average between six values sent every 10 times during the full hour. This allows the calculation of a more accurate estimation of the traffic of people in that specific time period. These values are then persisted in a database.
count = 0
while True:
#After five minutes restarts the count
if count == 5*60:
unpeople = json.dumps({"value" : sum_people, "TimeStamp" : time.time()})
client.publish(uniqueppl_topic, unpeople)
unvehicles = json.dumps({"value" : sum_vehicles, "TimeStamp" : time.time()})
client.publish(uniquevehicles_topic, unvehicles)
untwowheelers = json.dumps({"value" : sum_twowheelers, "TimeStamp" : time.time()})
client.publish(uniquetwowheelers_topic, untwowheelers)
sum_people = 0
sum_vehicles = 0
sum_twowheelers = 0
count = 0
people_warning = 0
vehicles_warning = 0
twowheelers_warning = 0
time.sleep(1) # sends values every second
count += 1
dif = (num_people - bk_numbers[0], num_vehicles - bk_numbers[1], num_twowheelers - bk_numbers[2])
if dif[0] > 0:
sum_people += num_people
if sum_people > 25 and people_warning == 0:
webhook.send("[Object Detection] More than 25 people!" + str(num_people))
people_warning = 1
if dif[1] > 0:
sum_vehicles += num_vehicles
if sum_vehicles > 50 and vehicles_warning == 0:
webhook.send("[Object Detection] More than 50 vehicles!" + str(num_vehicles))
vehicles_warning = 1
if dif[2] > 0:
sum_twowheelers += num_twowheelers
if sum_twowheelers > 25 and twowheelers_warning == 0:
webhook.send("[Object Detection] More than 25 twowheelers!" + str(num_twowheelers))
twowheelers_warning = 1
if not bk_numbers[0] == 0 or num_people != 0:
crpeople = json.dumps({"value" : num_people, "TimeStamp" : time.time()})
client.publish(currentppl_topic, crpeople)
if not bk_numbers[1] == 0 or num_vehicles != 0:
crvehicles = json.dumps({"value" : num_vehicles, "TimeStamp" : time.time()})
client.publish(currentvehicles_topic, crvehicles)
if not bk_numbers[2] == 0 or num_twowheelers != 0:
crtwowheelers = json.dumps({"value" : num_twowheelers, "TimeStamp" : time.time()})
client.publish(currenttwowheelers_topic, crtwowheelers)
bk_numbers = (num_people, num_vehicles, num_twowheelers)
All of the messages sent in topics to the local brokers are then sent to the central broker in IT to be accessed in the web application.
To run the program in the Jetson Nano:
python3 detector2.py rtsp://admin:admin@192.168.115.9/11
To continue running after closing the ssh session:
nohup python3 -u detector2.py rtsp://admin:admin@192.168.115.9/11 </dev/null >/dev/null 2>&1 &