This is really a useful and interesting article. Thank you very much.
That being said, I'm trying to run the project on a PC running Ubuntu 18.04.
I'm using a local registry for testing.
It seems that docker images were pushed to the local registry correctly:
$ docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
mcr.microsoft.com/azureiotedge-simulated-temperature-sensor 1.0 c86e0d919bd6 4 weeks ago 96.1MB
localhost:5000/image-classifier-service 1.1.91-amd64 3147a0658034 4 weeks ago 1.71GB
localhost:5000/camera-capture-opencv 1.1.91-amd64 cdcc320bd8a6 4 weeks ago 1.26GB
python 3.5 61bbcc36b492 5 weeks ago 909MB
...
mcr.microsoft.com/azureiotedge-agent 1.0 46ad173076af 2 months ago 137MB
mcr.microsoft.com/azureiotedge-diagnostics 1.0.8 d16965225a70 2 months ago 8.71MB
...
mcr.microsoft.com/azureiotedge-hub 1.0.7 ed05376f97bd 4 months ago 155MB
mcr.microsoft.com/azureiotedge-agent 1.0.7 219c2aff4adc 4 months ago 140MB
...
registry 2 f32a97de94e1 6 months ago 25.8MB
hello-world latest fce289e99eb9 8 months ago 1.84kB
However, deployment can't be completed. I found this in the edgeAgent logs:
2019-09-25 08:47:51.916 +00:00 [WRN] - Reconcile failed because of invalid configuration format
Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.ConfigFormatException: Agent configuration format is invalid. ---> System.ArgumentException: Image localhost:5000/camera-capture-opencv:1.1.91-amd64 is not in the right format
at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerConfig.ValidateAndGetImage(String image) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Docker/DockerConfig.cs:line 93
So it seems that there is a syntax error or something like that in the deployment file, but I can't find it. This file looks like this:
The problem is related to edgeAgent 1.0.7, as explained here.
After updating to 1.0.8, I can deploy the modules:
$ sudo iotedge list
NAME STATUS DESCRIPTION CONFIG
image-classifier-service running Up 11 minutes localhost:5000/image-classifier-service:1.1.91-amd64
edgeHub running Up 11 minutes mcr.microsoft.com/azureiotedge-hub:1.0.8
edgeAgent running Up 14 minutes mcr.microsoft.com/azureiotedge-agent:1.0.8
camera-capture running Up 11 minutes localhost:5000/camera-capture-opencv:1.1.91-amd64
Dream it. Build it. Retired Microsoft software engineer after 33 years. I explore AI agents, .NET, Python, C, microcontrollers and retro computing by building things and sharing what I learn.
Dave, I'm diving into your project to understand how it works in more detail.
First of all, I'm exploring the camera capture process. I'm running the project on a Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz.
I noticed that if the scene shot by the camera is still, no frames are processed. If the scene changes or if I move the camera, on average about 4 frames per second are processed:
$ iotedge logs camera-capture -f | ts %F-%H:%M:%.S
2019-09-27-10:42:31.697454 pygame 1.9.6
2019-09-27-10:42:31.697609 Hello from the pygame community. https://www.pygame.org/contribute.html
2019-09-27-10:42:31.697688 sasToken
2019-09-27-10:42:31.697758
2019-09-27-10:42:31.697833 Python 3.5.2 (default, Nov 12 2018, 13:43:14)
2019-09-27-10:42:31.697877 [GCC 5.4.0 20160609]
2019-09-27-10:42:31.697904
2019-09-27-10:42:31.697931 Camera Capture Azure IoT Edge Module. Press Ctrl-C to exit.
2019-09-27-10:42:31.697957 opening camera
...
2019-09-27-10:40:45.445710 sending frame to model: 476
2019-09-27-10:40:45.668826 label: Hand, probability 0.8052769303321838
2019-09-27-10:40:45.925346 sending frame to model: 477
2019-09-27-10:40:46.148182 label: Hand, probability 0.8468263745307922
2019-09-27-10:40:46.404615 sending frame to model: 478
2019-09-27-10:40:46.630166 label: Hand, probability 0.8512248992919922
2019-09-27-10:40:46.886933 sending frame to model: 479
2019-09-27-10:40:47.120413 label: Hand, probability 0.877470850944519
2019-09-27-10:40:47.377079 sending frame to model: 480
2019-09-27-10:40:47.601168 label: Hand, probability 0.8282925486564636
2019-09-27-10:40:47.857675 sending frame to model: 481
Did you achieve similar performances on your development host?
What about the RPi?
Dream it. Build it. Retired Microsoft software engineer after 33 years. I explore AI agents, .NET, Python, C, microcontrollers and retro computing by building things and sharing what I learn.
Hey yes, I did some optimisations, 1) if pixel change was greater that 70000 pixels RGB then send frame to ml model. 2) slowed down frame rate down, logic was the model would be more available to process a frame if something changed...
Hi, just wanted to let you know that I got your demo working on a Linux desktop x86_64 architecture, running Ubuntu 18.04.3, using a Logitech USB C922 webcam.
There were several challenges in finding all the things that needed tweaking, so I thought I'd share for others who may run into some of these issues.
1) deployment.template.json: Edit the azureSpeechServicesKey to match your Azure Cognitive Service's Speech service key (not BingKey, as stated in the tutorial)
2) module.json: In each module's folder, edit the "repository" line to point to your localhost:5000 instead of glovebox
3) azure_text_speech.py: Edit the TOKEN_URL to point to the one that Azure provides for you when you set up your speech service. Also edit the BASE_URL to point to the text-to-speech base URL for your region. For example, I had to edit mine to point to my region:
4) text2speech.py: For whatever reason, wf.getframerate() would not return the correct frame rate of my audio, causing an error.
Expression 'paInvalidSampleRate' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 2048
So I ran 'pacmd list-sinks' to find my actual audio sample rate (48000) and hardcoded it in place of wf.getframerate.
5) predict.py: Lastly, my camera-capture module kept getting connectivity issues, which was actually because it kept returning a response error stating:
Error: Could not preprocess image for prediction. module 'tensorflow' has no attribute 'Session'
This method was deprecated, so to fix this, edit the predict.py's line from 'tf.Session()' to 'tf.compat.v1.Session()'
After all was said and done, I was able to get it working:
Sure I could share my predict.py edit. It's simply a one line edit on line 123 from tf.Session() to tf.compat.v1.Session().
fromurllib.requestimporturlopenfromdatetimeimportdatetimeimporttimeimporttensorflowastffromPILimportImageimportnumpyasnpimportsysclassPredict():def__init__(self):self.filename='model.pb'self.labels_filename='labels.txt'self.network_input_size=0self.output_layer='loss:0'self.input_node='Placeholder:0'self.graph_def=tf.compat.v1.GraphDef()self.labels=[]self.graph=Noneself._initialize()def_initialize(self):print('Loading model...',end=''),withtf.io.gfile.GFile(self.filename,'rb')asf:self.graph_def.ParseFromString(f.read())tf.import_graph_def(self.graph_def,name='')self.graph=tf.compat.v1.get_default_graph()# Retrieving 'network_input_size' from shape of 'input_node'
input_tensor_shape=self.graph.get_tensor_by_name(self.input_node).shape.as_list()assertlen(input_tensor_shape)==4assertinput_tensor_shape[1]==input_tensor_shape[2]self.network_input_size=input_tensor_shape[1]withopen(self.labels_filename,'rt')aslf:self.labels=[l.strip()forlinlf.readlines()]def_log_msg(self,msg):print("{}: {}".format(time.time(),msg))def_resize_to_256_square(self,image):w,h=image.sizenew_w=int(256/h*w)image.thumbnail((new_w,256),Image.ANTIALIAS)returnimagedef_crop_center(self,image):w,h=image.sizexpos=(w-self.network_input_size)/2ypos=(h-self.network_input_size)/2box=(xpos,ypos,xpos+self.network_input_size,ypos+self.network_input_size)returnimage.crop(box)def_resize_down_to_1600_max_dim(self,image):w,h=image.sizeifh<1600andw<1600:returnimagenew_size=(1600*w//h,1600)if(h>w)else(1600,1600*h//w)self._log_msg("resize: "+str(w)+"x"+str(h)+" to "+str(new_size[0])+"x"+str(new_size[1]))ifmax(new_size)/max(image.size)>=0.5:method=Image.BILINEARelse:method=Image.BICUBICreturnimage.resize(new_size,method)def_convert_to_nparray(self,image):# RGB -> BGR
image=np.array(image)returnimage[:,:,(2,1,0)]def_update_orientation(self,image):exif_orientation_tag=0x0112ifhasattr(image,'_getexif'):exif=image._getexif()ifexif!=Noneandexif_orientation_taginexif:orientation=exif.get(exif_orientation_tag,1)self._log_msg('Image has EXIF Orientation: '+str(orientation))# orientation is 1 based, shift to zero based and flip/transpose based on 0-based values
orientation-=1iforientation>=4:image=image.transpose(Image.TRANSPOSE)iforientation==2ororientation==3ororientation==6ororientation==7:image=image.transpose(Image.FLIP_TOP_BOTTOM)iforientation==1ororientation==2ororientation==5ororientation==6:image=image.transpose(Image.FLIP_LEFT_RIGHT)returnimagedefpredict_url(self,imageUrl):self._log_msg("Predicting from url: "+imageUrl)withurlopen(imageUrl)astestImage:image=Image.open(testImage)returnself.predict_image(image)defpredict_image(self,image):try:ifimage.mode!="RGB":self._log_msg("Converting to RGB")image=image.convert("RGB")# Update orientation based on EXIF tags
image=self._update_orientation(image)image=self._resize_down_to_1600_max_dim(image)image=self._resize_to_256_square(image)image=self._crop_center(image)cropped_image=self._convert_to_nparray(image)withself.graph.as_default():withtf.compat.v1.Session()assess:prob_tensor=sess.graph.get_tensor_by_name(self.output_layer)predictions,=sess.run(prob_tensor,{self.input_node:[cropped_image]})result=[]forp,labelinzip(predictions,self.labels):truncated_probablity=np.float64(round(p,8))iftruncated_probablity>1e-8:result.append({'tagName':label,'probability':truncated_probablity,'tagId':'','boundingBox':None})print('[%s]'%', '.join(map(str,result)))response={'id':'','project':'','iteration':'','created':datetime.utcnow().isoformat(),'predictions':result}returnresponseexceptExceptionase:self._log_msg(str(e))return'Error: Could not preprocess image for prediction. '+str(e)
Hi ,
This article is helpful.
How can i make the same image classification module without raspberry pi and on my Ubuntu platform.
Any help would be great.
Dream it. Build it. Retired Microsoft software engineer after 33 years. I explore AI agents, .NET, Python, C, microcontrollers and retro computing by building things and sharing what I learn.
Yes absolutely. I mostly built the project on Ubuntu 18.04 on my laptop and then ported to Raspberry Pi. You will see there are Dockerfiles for x86 in the project. Cheers Dave
Dream it. Build it. Retired Microsoft software engineer after 33 years. I explore AI agents, .NET, Python, C, microcontrollers and retro computing by building things and sharing what I learn.
On the bottom bar of Visual Studio Code there is the option to switch the project from armv32 to amd64. That is how you build the containers for arm64. The project will work with most USB cameras and the camera module is using OpenCV to capture frames from the USB camera.
Thank you very much, you've saved me countless hours over the last couple days while I've been learning Azure IoT.
Your Dockerfile for OpenCV are also a life saver (i found the ones in the Azure-Samples github.com/Azure-Samples/Custom-vi... to fail to build properly)
HI,need a little help.I am receiving this in my camera capture log. Camera is not opening.I am using Ubuntu and not raspberry pi. Following is my code:
Camera Capture Azure IoT Edge Module. Press Ctrl-C to exit.
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib confmisc.c:1286:(snd_func_refer) Unable to find definition 'cards.ICH.pcm.surround71.0:CARD=0'
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4771:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM surround71
ALSA lib setup.c:548:(add_elem) Cannot obtain info for CTL elem (MIXER,'IEC958 Playback Default',0,0,0): No such file or directory
ALSA lib setup.c:548:(add_elem) Cannot obtain info for CTL elem (MIXER,'IEC958 Playback Default',0,0,0): No such file or directory
ALSA lib setup.c:548:(add_elem) Cannot obtain info for CTL elem (MIXER,'IEC958 Playback Default',0,0,0): No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
All of those "ALSA lib" lines have nothing to do with your camera, those are a result of your audio device, so you can ignore them as they are just warnings.
Is there any LED light on your camera that light up when the camera-capture module is running? For example, I have a C922 Logitech webcam, and on the device you can see the white LED lights turn on when camera-capture module starts running.
If you see no lights, or suspect camera-capture module isn't connecting to your camera, then most likely it means your camera device index doesn't match what is in your deployment.template.json file. For example, my webcam shows up as "/dev/video0" and "/dev/video1" whenever I plug it in. Find yours by opening up a terminal window, unplug your camera, type "ls /dev/video*" and see what shows up, then plug in your camera and type "ls /dev/video*" again to determine the number of your camera index. Then in your deployment.template.json, edit the "PathOnHost" and "PathInContainer" parameters to match your device.
Dream it. Build it. Retired Microsoft software engineer after 33 years. I explore AI agents, .NET, Python, C, microcontrollers and retro computing by building things and sharing what I learn.
Dream it. Build it. Retired Microsoft software engineer after 33 years. I explore AI agents, .NET, Python, C, microcontrollers and retro computing by building things and sharing what I learn.
Dream it. Build it. Retired Microsoft software engineer after 33 years. I explore AI agents, .NET, Python, C, microcontrollers and retro computing by building things and sharing what I learn.
Hey there - as Tom pointed out about the ALSA messages are to do with audio and are just warnings. I think there must be an issue with your USB camera and OpenCV. What camera are you using? Cheers Dave
We're a place where coders share, stay up-to-date and grow their careers.
This is really a useful and interesting article. Thank you very much.
That being said, I'm trying to run the project on a PC running Ubuntu 18.04.
I'm using a local registry for testing.
It seems that docker images were pushed to the local registry correctly:
However, deployment can't be completed. I found this in the edgeAgent logs:
So it seems that there is a syntax error or something like that in the deployment file, but I can't find it. This file looks like this:
Any help would be greatly appreciated.
The problem is related to edgeAgent 1.0.7, as explained here.
After updating to 1.0.8, I can deploy the modules:
Ah, fantastic. Thanks and great you got working. I'll update the deployment template so it starts with 1.0.8. Let me know how you get on. Cheers Dave
Dave, I'm diving into your project to understand how it works in more detail.
First of all, I'm exploring the camera capture process. I'm running the project on a Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz.
I noticed that if the scene shot by the camera is still, no frames are processed. If the scene changes or if I move the camera, on average about 4 frames per second are processed:
Did you achieve similar performances on your development host?
What about the RPi?
I've just found this in
CameraCapture.py...It answers my question ;)
Hey yes, I did some optimisations, 1) if pixel change was greater that 70000 pixels RGB then send frame to ml model. 2) slowed down frame rate down, logic was the model would be more available to process a frame if something changed...
Hi, just wanted to let you know that I got your demo working on a Linux desktop x86_64 architecture, running Ubuntu 18.04.3, using a Logitech USB C922 webcam.
There were several challenges in finding all the things that needed tweaking, so I thought I'd share for others who may run into some of these issues.
1) deployment.template.json: Edit the azureSpeechServicesKey to match your Azure Cognitive Service's Speech service key (not BingKey, as stated in the tutorial)
2) module.json: In each module's folder, edit the "repository" line to point to your localhost:5000 instead of glovebox
3) azure_text_speech.py: Edit the TOKEN_URL to point to the one that Azure provides for you when you set up your speech service. Also edit the BASE_URL to point to the text-to-speech base URL for your region. For example, I had to edit mine to point to my region:
4) text2speech.py: For whatever reason, wf.getframerate() would not return the correct frame rate of my audio, causing an error.
Expression 'paInvalidSampleRate' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 2048
So I ran 'pacmd list-sinks' to find my actual audio sample rate (48000) and hardcoded it in place of wf.getframerate.
5) predict.py: Lastly, my camera-capture module kept getting connectivity issues, which was actually because it kept returning a response error stating:
Error: Could not preprocess image for prediction. module 'tensorflow' has no attribute 'Session'
This method was deprecated, so to fix this, edit the predict.py's line from 'tf.Session()' to 'tf.compat.v1.Session()'
After all was said and done, I was able to get it working:
Hi ,
Thanks for sharing.I am receiving the same error of your 5 point. In my predict.py file "tf.Session" is missing. Any help will be great.
if possible could u share your this predict.py file.
Sure I could share my predict.py edit. It's simply a one line edit on line 123 from tf.Session() to tf.compat.v1.Session().
Hi ,
This article is helpful.
How can i make the same image classification module without raspberry pi and on my Ubuntu platform.
Any help would be great.
Yes absolutely. I mostly built the project on Ubuntu 18.04 on my laptop and then ported to Raspberry Pi. You will see there are Dockerfiles for x86 in the project. Cheers Dave
Thanks .
Also do u have any document for connecting a physical device like camera to this project.
On the bottom bar of Visual Studio Code there is the option to switch the project from armv32 to amd64. That is how you build the containers for arm64. The project will work with most USB cameras and the camera module is using OpenCV to capture frames from the USB camera.
Thanks a lot sir for your efforts.
Also , do i have to delete the arm 32v7 files and from platform also if i am not using raspberry pi.
Thanks a lot again.
Thank you very much, you've saved me countless hours over the last couple days while I've been learning Azure IoT.
Your Dockerfile for OpenCV are also a life saver (i found the ones in the Azure-Samples github.com/Azure-Samples/Custom-vi... to fail to build properly)
p.s. nice surname.
HI,need a little help.I am receiving this in my camera capture log. Camera is not opening.I am using Ubuntu and not raspberry pi. Following is my code:
Camera Capture Azure IoT Edge Module. Press Ctrl-C to exit.
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib confmisc.c:1286:(snd_func_refer) Unable to find definition 'cards.ICH.pcm.surround71.0:CARD=0'
ALSA lib conf.c:4292:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:4771:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM surround71
ALSA lib setup.c:548:(add_elem) Cannot obtain info for CTL elem (MIXER,'IEC958 Playback Default',0,0,0): No such file or directory
ALSA lib setup.c:548:(add_elem) Cannot obtain info for CTL elem (MIXER,'IEC958 Playback Default',0,0,0): No such file or directory
ALSA lib setup.c:548:(add_elem) Cannot obtain info for CTL elem (MIXER,'IEC958 Playback Default',0,0,0): No such file or directory
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
@heldss
All of those "ALSA lib" lines have nothing to do with your camera, those are a result of your audio device, so you can ignore them as they are just warnings.
Is there any LED light on your camera that light up when the camera-capture module is running? For example, I have a C922 Logitech webcam, and on the device you can see the white LED lights turn on when camera-capture module starts running.
If you see no lights, or suspect camera-capture module isn't connecting to your camera, then most likely it means your camera device index doesn't match what is in your deployment.template.json file. For example, my webcam shows up as "/dev/video0" and "/dev/video1" whenever I plug it in. Find yours by opening up a terminal window, unplug your camera, type "ls /dev/video*" and see what shows up, then plug in your camera and type "ls /dev/video*" again to determine the number of your camera index. Then in your deployment.template.json, edit the "PathOnHost" and "PathInContainer" parameters to match your device.
Restart your camera-capture from Terminal with "iotedge restart camera-capture", and see if it works.
Very helpful tutorial and relevant to my current work.
Hey, that is awesome and great that you find helpful.
Would you mind telling me how you found the posting - did you come to dev.to and what did you search for as I was not sure how to tag this post.
Cheers and thanks Dave
I follow you on twitter and came across this post in twitter updates. Also I came to know about dev.to only through your post.
ah cool - thanks for the follow. Feel free to ask any questions. Cheers Dave
Hi,
I am facing this issue.
Camera Capture Azure IoT Edge Module. Press Ctrl-C to exit.
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm_route.c:867:(find_matching_chmap) Found no matching channel map
Hey there - as Tom pointed out about the ALSA messages are to do with audio and are just warnings. I think there must be an issue with your USB camera and OpenCV. What camera are you using? Cheers Dave