Question about "Create an AI companion robot" | Viam Robotics. | Page 1

gilded flare Apr 19, 2023, 2:08 AM

#

Hi, I'm on step #1 of the Rover setup in this tutorial. It says, "Wire your servo to the Pi by attaching the black wire to ground, red wire to an available 5V pin, and yellow wire to pin 8." So pins 4 & 6 were being used already by the rover so I moved the 5v wire to pin 2 and the ground wire to pin 14 which the viam rover needs. Then I realized that the instructions were off. The servo's pin wires come attached to each other and can't be separated. The 5v and ground wires are switched so they can't fit in pins 4, 6, and 8 as instructed. Also the instructions say to wire the yellow wire to pin 8 and the servo doesn't have a yellow wire.

vague storm Apr 19, 2023, 2:28 PM

#

hey!! That is tricky. Do you have access to male-female jumper wires, like these: https://www.digikey.com/en/products/detail/sparkfun-electronics/PRT-12794/5993859?utm_adgroup=Jumper Wire&utm_source=google&utm_medium=cpc&utm_campaign=Shopping_Product_Prototyping%2C Fabrication Products_NEW&utm_term=&utm_content=Jumper Wire&gclid=CjwKCAjwov6hBhBsEiwAvrvN6CWCxa5b4vK3wbrT94NNrhgPeBxMd5qZuPbyYJay-ybfTs_WATyiqRoCJpcQAvD_BwE ? You can connect these to your servo wires (as shown in the attached image), and then connect the extended wires to 5v, ground, and GPIO on your Pi

gilded flare Apr 19, 2023, 2:31 PM

#

That’s a good solution, i should have some! Let me try that and see how that works out

vague storm Apr 20, 2023, 2:16 PM

#

Awesome!

gilded flare Apr 20, 2023, 6:33 PM

#

vague storm Awesome!

I believe I was able to wire it correctly but I can't test since when I try running my run.sh, I get this output:

viampi@viampi:~/tutorial-openai-integration $ ./run.sh
pygame 2.1.2 (SDL 2.0.16, Python 3.9.2)
Hello from the pygame community. https://www.pygame.org/contribute.html
Traceback (most recent call last): File "/home/viampi/tutorial-openai-integration/rosey.py", line 6, in <module> from elevenlabs import ElevenLabs
ImportError: cannot import name 'ElevenLabs' from 'elevenlabs' (/home/viampi/.local/lib/python3.9/site-packages/elevenlabs/__init__.py)

I checked and it says the ElevenLabs requirement was already satisfied:
Requirement already satisfied: typing-extensions in /home/viampi/.local/lib/python3.9/site-packages (from ipython>=8.0->elevenlabs->-r requirements.txt (line 23)) (4.5.0)

#

Also for this part of the tutorial:

#

#

I believe I set it up correctly:

#

#

But it's not under control:

#

#

Logs are showing this error but I don't see where I can input the name of the board:
2023-04-20T18:36:21.945Z error robot_server impl/resource_manager.go:403 component config validation error resource rdk:component:servo/servo1 model rdk:builtin:pi error error validating "": need the name of the board

uneven trout Apr 20, 2023, 7:41 PM

#

can you try changing the servo configuration from

attributes": {
        "pin": "8"
}

to

attributes": {
        "pin": "8",
        "board": "local"
}

#

I've managed to reproduce the elevenlabs python error. Will do some investigating as well 😄

#

I think that the API for that package has changed.

Can you try changing

from elevenlabs import ElevenLabs

to

import elevenlabs as eleven

and then instead of

eleven = ElevenLabs(params.elevenlabs_key)

this

eleven.set_api_key(params.elevenlabs_key)

Let me know if that works? I don't have stable internet connection right now so can't test unfortunately

gilded flare Apr 20, 2023, 8:12 PM

#

uneven trout can you try changing the servo configuration from ``` attributes": { "p...

The Servo works now! Thanks!

gilded flare Apr 20, 2023, 8:13 PM

#

uneven trout I *think* that the API for that package has changed. Can you try changing ``` ...

Just tried that, here's the new error output:

viampi@viampi:~/tutorial-openai-integration $ ./run.sh
pygame 2.1.2 (SDL 2.0.16, Python 3.9.2)
Hello from the pygame community. https://www.pygame.org/contribute.html
2023-04-20 16:11:44,775 ERROR viam (init.py:34) [ERROR] Uncaught exception
Traceback (most recent call last):
File "/home/viampi/tutorial-openai-integration/rosey.py", line 24, in <module>
con = sqlite3.connect("cache/rosey_cache.db")
sqlite3.OperationalError: unable to open database file
Traceback (most recent call last):
File "/home/viampi/tutorial-openai-integration/rosey.py", line 24, in <module>
con = sqlite3.connect("cache/rosey_cache.db")
sqlite3.OperationalError: unable to open database file

uneven trout Apr 20, 2023, 8:22 PM

#

Yeah I think I see what happened here. The author added caching with a sqlite database but didn't update the instructions... I'm sorry.

You should be able to delete the lines relating to that without any issue though as far as I can see.

In my file that's lines 23-28 and 101-104 and 119-122

#

so that's

# set up db cache
con = sqlite3.connect("cache/rosey_cache.db")
cur = con.cursor()
res = cur.execute("SELECT name FROM sqlite_master WHERE name='responses'")
if res.fetchone() is None:
    cur.execute("CREATE TABLE responses(prompt, char, mood, response)")

#

    # get cached response if it exists
    res = cur.execute("SELECT response FROM responses WHERE prompt=? and char=? and mood=?", (command, current_char, current_mood))
    response = res.fetchone()
    if(response):
        return response[0]

#

and

        if(not re.match("^As an AI language model", completion)):
            # cache this completion
            cur.execute("INSERT into responses VALUES(?, ?, ?, ?)", (command, current_char, current_mood, completion))
            con.commit()

gilded flare Apr 20, 2023, 8:52 PM

#

📎 message.txt

gilded flare Apr 20, 2023, 8:53 PM

#

uneven trout Yeah I think I see what happened here. The author added caching with a sqlite da...

That's ok, I Just finished removing those lines. Here's the new output

uneven trout Apr 20, 2023, 9:04 PM

#

Oh dear that i don't know what to do with either @small kindle are you around?

gilded flare Apr 20, 2023, 9:04 PM

#

Tried it again and got a different output

📎 message.txt

gilded flare Apr 20, 2023, 9:04 PM

#

uneven trout Oh dear that i don't know what to do with either <@1083785069962993684> are you...

That's okay, thanks for all the help!

small kindle Apr 20, 2023, 9:10 PM

#

those look to me like errors from your python script about ALSA not being configured properly. are you doing audio things in your script? is audio set up on your pi?

gilded flare Apr 20, 2023, 9:17 PM

#

Latest try for run.sh

📎 message.txt

gilded flare Apr 20, 2023, 9:17 PM

#

small kindle those look to me like errors from your python script about ALSA not being config...

I'm just trying to do this project:
https://docs.viam.com/tutorials/projects/integrating-viam-with-openai/

Viam Documentation

Integrate Viam with ChatGPT to create a companion robot

Harness AI to add life to your Viam rover.

#

I'm following all the steps in the tutorial

small kindle Apr 20, 2023, 9:33 PM

#

unfortunately, i'm not familiar with that particular tutorial. did you install alsa-tools and alsa-utils?

#

does the commandline alsamixer tool allow you to adjust the volume of your device?

#

unfortunately the employee who wrote this is on vacation. he's the true expert here and once he is back, i'll make sure he hops on to work with you on this problem. sorry for the inconvenience

gilded flare Apr 20, 2023, 9:56 PM

#

Most likely I don’t have them installed but i can check when i get back to my desk. But no worries, thanks for the help. Do you estimation on when he’ll be back?

small kindle Apr 20, 2023, 9:56 PM

#

i believe he gets back next week

grave pier Apr 24, 2023, 2:10 PM

#

Hi @gilded flare - sorry for the delay here, as my colleagues noted, I was on vacation. So first, with the sqlite - sqlite3 is in fact built-in to python 3. However, an empty cache/ directory was missing in the tutorial repository, so the error you were getting was in regards to trying to write in a location that did not exist. I just added that empty directory, so that caching should now work again if you pull and use the original un-altered script. That said, the caching is pretty basic so you could extend it with your own caching strategy to vary responses, etc.

#

Now, as for the other errors - unfortunately, I have always seen a number of ALSA and jack errors output from this when actually working normally. So, as ugly as they are, I think they are benign and in this case are a red herring. I think the real issue here is connecting to your robot via this script.
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
Exception ignored in: <function ViamChannel.del at 0x7f92bb9e50>
Traceback (most recent call last):
File "/home/viampi/.local/lib/python3.9/site-packages/viam/rpc/dial.py", line 146, in del
File "/home/viampi/.local/lib/python3.9/site-packages/viam/rpc/dial.py", line 141, in close
File "/home/viampi/.local/lib/python3.9/site-packages/grpclib/client.py", line 787, in close
File "/home/viampi/.local/lib/python3.9/site-packages/grpclib/protocol.py", line 554, in close
File "/home/viampi/.local/lib/python3.9/site-packages/grpclib/protocol.py", line 234, in close
File "/usr/lib/python3.9/asyncio/selector_events.py", line 700, in close
File "/usr/lib/python3.9/asyncio/base_events.py", line 746, in call_soon
File "/usr/lib/python3.9/asyncio/base_events.py", line 510, in _check_closed
RuntimeError: Event loop is closed

#

Those appear to be coming from the viam sdk

#

If you run the sample python script from your "code sample" tab for your rover, does that work correctly?

#

If so, can you ensure you are setting VIAM_SECRET and VIAM_ADDRESS in your run.sh with the values from lines 18 and 23 from the code sample?

gilded flare Apr 24, 2023, 2:43 PM

#

grave pier If so, can you ensure you are setting VIAM_SECRET and VIAM_ADDRESS in your run.s...

Unfortunately I won’t have access to my viam rover until later this evening but I’ll be able to update you then!

grave pier Apr 24, 2023, 2:53 PM

#

Sounds good! Also, I know @uneven trout was able to help with the elevenlabs update - I will update the tutorial code as well.

gilded flare Apr 25, 2023, 8:08 PM

#

grave pier Sounds good! Also, I know <@254751870768119809> was able to help with the eleve...

Hi Matt, I pulled the original un-altered script as you suggested and saw the changes you updated.

gilded flare Apr 25, 2023, 8:11 PM

#

grave pier If so, can you ensure you are setting VIAM_SECRET and VIAM_ADDRESS in your run.s...

Yes, I've double-checked that they are the same values. I'd be happy to private message them to you to verify I'm using the correct values and not a possible mistake I made.

#

Current try of running the script:

This is the live logs:

📎 message.txt

#

This is the output from the pi:

📎 message.txt

gilded flare Apr 25, 2023, 8:44 PM

#

grave pier If you run the sample python script from your "code sample" tab for your rover, ...

For this, I'm not sure. I assume you mean running it on another system like installing viam server on a mac or linux machine? I'm installing it now to my mac and can test that after it's completed installation.

gilded flare Apr 25, 2023, 9:21 PM

#

I was able to use my mac as a viam-server host. But I’m assuming this is not applicable to this scenario.

gilded flare Apr 25, 2023, 9:40 PM

#

I have the sdk installed on my mac

#

This is new territory for me, but I'll keep trying

#

grave pier Apr 25, 2023, 11:15 PM

#

gilded flare For this, I'm not sure. I assume you mean running it on another system like inst...

No, I meant running the code sample on your rover pi

gilded flare Apr 25, 2023, 11:27 PM

#

grave pier No, I meant running the code sample on your rover pi

Okay that makes sense. I’ll try that tomorrow morning

gilded flare Apr 26, 2023, 1:58 PM

#

grave pier No, I meant running the code sample on your rover pi

Traceback (most recent call last):
File "/home/viampi/viam.py", line 3, in <module>
from viam.robot.client import RobotClient
File "/home/viampi/viam.py", line 3, in <module>
from viam.robot.client import RobotClient
ModuleNotFoundError: No module named 'viam.robot'; 'viam' is not a package

I''m getting this error after running the code

grave pier Apr 26, 2023, 3:32 PM

#

hm, maybe try re-installing or upgrading the Viam SDK? https://python.viam.dev/

gilded flare Apr 26, 2023, 3:40 PM

#

No luck with that unfortunately:
viampi@viampi:~ $ pip install viam-sdk
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: viam-sdk in ./.local/lib/python3.9/site-packages (0.2.10)
Requirement already satisfied: grpclib<0.5.0,>=0.4.3 in ./.local/lib/python3.9/site-packages (from viam-sdk) (0.4.3)
Requirement already satisfied: Pillow<10.0.0,>=9.3.0 in ./.local/lib/python3.9/site-packages (from viam-sdk) (9.4.0)
Requirement already satisfied: googleapis-common-protos<2.0.0,>=1.57.0 in ./.local/lib/python3.9/site-packages (from viam-sdk) (1.58.0)
Requirement already satisfied: typing-extensions<5.0.0,>=4.4.0 in ./.local/lib/python3.9/site-packages (from viam-sdk) (4.5.0)
Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 in ./.local/lib/python3.9/site-packages (from googleapis-common-protos<2.0.0,>=1.57.0->viam-sdk) (4.22.0)
Requirement already satisfied: h2<5,>=3.1.0 in ./.local/lib/python3.9/site-packages (from grpclib<0.5.0,>=0.4.3->viam-sdk) (4.1.0)
Requirement already satisfied: multidict in ./.local/lib/python3.9/site-packages (from grpclib<0.5.0,>=0.4.3->viam-sdk) (6.0.2)
Requirement already satisfied: hyperframe<7,>=6.0 in ./.local/lib/python3.9/site-packages (from h2<5,>=3.1.0->grpclib<0.5.0,>=0.4.3->viam-sdk) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in ./.local/lib/python3.9/site-packages (from h2<5,>=3.1.0->grpclib<0.5.0,>=0.4.3->viam-sdk) (4.0.0)
viampi@viampi:~ $ pip install -U viam-sdk
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: viam-sdk in ./.local/lib/python3.9/site-packages (0.2.10)
Collecting viam-sdk
Downloading viam_sdk-0.2.18-py3-none-manylinux2014_aarch64.whl (9.2 MB)
|████████████████████████████████| 9.2 MB 61 kB/s
Collecting protobuf<5.0.0,>=4.22.1
Downloading protobuf-4.22.3-cp37-abi3-manylinux2014_aarch64.whl (301 kB)
|████████████████████████████████| 301 kB 3.6 MB/s
Requirement already satisfied: googleapis-common-protos<2.0.0,>=1.58.0 in ./.local/lib/python3.9/site-packages (from viam-sdk) (1.58.0)
Requirement already satisfied: Pillow<10.0.0,>=9.4.0 in ./.local/lib/python3.9/site-packages (from viam-sdk) (9.4.0)
Requirement already satisfied: typing-extensions<5.0.0,>=4.5.0 in ./.local/lib/python3.9/site-packages (from viam-sdk) (4.5.0)
Requirement already satisfied: grpclib<0.5.0,>=0.4.3 in ./.local/lib/python3.9/site-packages (from viam-sdk) (0.4.3)
Requirement already satisfied: multidict in ./.local/lib/python3.9/site-packages (from grpclib<0.5.0,>=0.4.3->viam-sdk) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in ./.local/lib/python3.9/site-packages (from grpclib<0.5.0,>=0.4.3->viam-sdk) (4.1.0)
Requirement already satisfied: hyperframe<7,>=6.0 in ./.local/lib/python3.9/site-packages (from h2<5,>=3.1.0->grpclib<0.5.0,>=0.4.3->viam-sdk) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in ./.local/lib/python3.9/site-packages (from h2<5,>=3.1.0->grpclib<0.5.0,>=0.4.3->viam-sdk) (4.0.0)
Installing collected packages: protobuf, viam-sdk
Attempting uninstall: protobuf
Found existing installation: protobuf 4.22.0
Uninstalling protobuf-4.22.0:
Successfully uninstalled protobuf-4.22.0
Attempting uninstall: viam-sdk
Found existing installation: viam-sdk 0.2.10
Uninstalling viam-sdk-0.2.10:
Successfully uninstalled viam-sdk-0.2.10
Successfully installed protobuf-4.22.3 viam-sdk-0.2.18
viampi@viampi:~ $ python viam.py
Traceback (most recent call last):
File "/home/viampi/viam.py", line 3, in <module>
from viam.robot.client import RobotClient
File "/home/viampi/viam.py", line 3, in <module>
from viam.robot.client import RobotClient
ModuleNotFoundError: No module named 'viam.robot'; 'viam' is not a package

grave pier Apr 26, 2023, 5:16 PM

#

hm... @small kindle - any ideas here? This seems to be an issue with viam sdk, not the AI project itself

grave pier Apr 26, 2023, 7:41 PM

#

@gilded flare try renaming your file from viam.py to something else

gilded flare Apr 26, 2023, 9:19 PM

#

grave pier <@268191595780767745> try renaming your file from viam.py to something else

📎 message.txt

grave pier Apr 27, 2023, 2:49 PM

#

In the above, it appears you are connecting correctly and most is working, other than your camera.

#

Is your camera working correctly via the Control tab in the viam app?

gilded flare Apr 27, 2023, 2:51 PM

#

Yea i’ve been having issues with the camera since i received the rover

#

It's not

#

This is the error I get:
2023-04-27T15:00:27.848Z error robot_server impl/resource_manager.go:549 error building resource resource rdk:service:vision/vision model rdk:builtin:builtin error unknown resource type: rdk:service:vision and/or model: rdk:builtin:builtin
2023-04-27T15:00:27.602Z error robot_server.rdk:component:camera/cam videosource/webcam.go:475 failed to reconnect camera error failed to find camera: cannot open webcam: failed to find the best driver that fits the constraints

grave pier Apr 27, 2023, 3:06 PM

#

hm, if you try leaving the path blank for your camera config, I'd be curious if that helps

#

It should then auto-detect the path, even if it changes

#

I am not sure if that is a separate problem from the issue running the rosey script - but it would cause issues with the rosey script regardless

gilded flare Apr 27, 2023, 3:42 PM

#

Looks like it work! Thanks!

#

Tried running the script again, I heard the servo spin for a second and is stuck here

📎 message.txt

#

Live feed is still functional from the control tab when enabled

#

2023-04-27T15:44:09.599Z error robot_server impl/resource_manager.go:549 error building resource resource rdk:service:vision/vision model rdk:builtin:builtin error unknown resource type: rdk:service:vision and/or model: rdk:builtin:builtin

grave pier Apr 27, 2023, 4:59 PM

#

My apologies, but you may now be hitting a breaking vision service change:
https://docs.viam.com/appendix/release-notes/

Viam Documentation

Release Notes

25 April 2023 Breaking Changes Vision Service The Vision Service is becoming more modular in RDK v0.2.36, API v0.1.118, and Python SDK v0.2.18.
The following breaking changes will take effect:
Use individual vision service instances Add and remove models using the robot config Add machine learning vision models to a vision service Use individual...

#

I have yet to update the tutorial to use the vision service in the new way. so that might now be the issue

gilded flare Apr 27, 2023, 5:01 PM

#

Oh okay, should I revert any changes or just wait for your update?

grave pier Apr 27, 2023, 5:14 PM

#

nothing to revert - you can try to follow steps as per the docs I linked above, or wait - I will try to update tomorrow or latest Monday.

gilded flare May 1, 2023, 8:54 PM

#

grave pier nothing to revert - you can try to follow steps as per the docs I linked above, ...

Hi, just following up. Any eta on when you think you're able to do the update or it shouldn't be an issue for me to do myself? I'm a beginner to this so I don't mind trying to figure it out but I'm not sure what to modify as per the docs

grave pier May 2, 2023, 12:41 PM

#

Hi @gilded flare - sorry, I have not had a chance - but you should be able to follow the steps yourself from the docs. If you get stuck let me know, and I can at some point put in the updates this week.

gilded flare May 2, 2023, 2:16 PM

#

grave pier Hi <@268191595780767745> - sorry, I have not had a chance - but you should be ab...

No worries, I tried to follow the steps in the doc

Line 139:
Old: service = VisionServiceClient.from_robot(robot, 'vision')
New: vision = VisionServiceClient.from_robot(robot)

Line 145:
Old: detections = await service.get_classifications_from_camera(camera_name='cam', classifier_name='stuff_classifier', count=1)
New: detections = await my_object_detector.get_detections(img)

Are these the only lines I have to update in rosey.py?

grave pier May 3, 2023, 2:28 PM

#

I think so, but then you need to update your config as well

gilded flare May 3, 2023, 3:13 PM

#

Here is my config raw JSON in it's current state:

📎 message.txt

#

I also enabled the Viam support option

gilded flare May 3, 2023, 4:41 PM

#

Do i only modify the vision services? There’s two for some reason, they look the same so i assume they’re conflicting?

grave pier May 3, 2023, 6:56 PM

#

You need to now use an mlmodel service and a vision service. My relevant config looks like this:
{
"model": "tflite_cpu",
"attributes": {
"model_path": "/home/mcvella/lite-model_efficientdet_lite1_detection_metadata_1.tflite",
"label_path": "/home/mcvella/labelmap.txt",
"num_threads": 1
},
"name": "stuff-detector",
"type": "mlmodel"
},
{
"type": "vision",
"model": "mlmodel",
"attributes": {
"mlmodel_name": "stuff-detector"
},
"name": "vis-stuff-detector"
}

#

note that I switched to a detector (instead of classifier) but the config should look about the same

gilded flare May 3, 2023, 7:37 PM

#

{
"name": "stuff-detector",
"type": "mlmodel",
"model": "tflite_cpu",
"attributes": {
"model_path": "/home/viampi/lite-model_efficientdet_lite1_detection_metadata_1.tflite",
"label_path": "/home/viampi/labelmap.txt",
"num_threads": "1"
}
},
{
"name": "vis-stuff-detector",
"type": "vision",
"model": "mlmodel",
"attributes": {
"mlmodel_name": "stuff-detector"
}

I added the services as you showed above.

gilded flare May 3, 2023, 7:37 PM

#

gilded flare No worries, I tried to follow the steps in the doc --------- Line 139: Old: serv...

I didn't make this change, is it needed or is it incorrect?

#

Here's the current state of my raw JSON config:

📎 message.txt

gilded flare May 3, 2023, 8:30 PM

#

I'm pretty sure I do not have a tflite model on uploaded. I'm not aware of a way to check on the pi but if it's supposed to be on-board then I'm not sure how to check.

#

@grave pier If you have a schedule opening sometime tomorrow or friday, I'd appreciate it if we can get on a call because I'm not sure where I'm going wrong.

grave pier May 4, 2023, 4:38 AM

#

@gilded flare - I will get back to you on this tomorrow, thanks

gilded flare May 4, 2023, 3:15 PM

#

grave pier <@268191595780767745> - I will get back to you on this tomorrow, thanks

Thanks, I appreciate it

grave pier May 5, 2023, 12:34 AM

#

Hi Ali, this is not officially published yet, but you can preview the needed config changes here: https://docs-test.viam.dev/4fb3b5acee30792c701a845a715e3d150f564721/public/tutorials/projects/integrating-viam-with-openai/#2-configure-the-ml-model-and-vision-services-to-use-the-classifier - sorry for the delay

Viam Documentation

Integrate Viam with ChatGPT to create a companion robot

Harness AI to add life to your Viam rover.

gilded flare May 5, 2023, 1:17 PM

#

grave pier Hi Ali, this is not officially published yet, but you can preview the needed con...

Thanks! I'll get started on it

gilded flare May 5, 2023, 2:00 PM

#

@grave pier

#

Everything looks correct as per the docs

#

I redid the tutorial and this was my output after running the script:

#

📎 message.txt

grave pier May 5, 2023, 2:08 PM

#

That appears to be an issue with connecting to Viam server running on your robot again... Can you again verify that running the python from the code sample tab in the app from your pi works as expected?

gilded flare May 5, 2023, 2:08 PM

#

grave pier That appears to be an issue with connecting to Viam server running on your robot...

Does not look like it does

#

📎 message.txt

#

Looks like a similar issue with the vision service?

#

2023-05-05T14:10:26.900Z error robot_server impl/resource_manager.go:566 error building resource resource rdk:service:vision/vision model rdk:builtin:builtin error unknown resource type: rdk:service:vision and/or model: rdk:builtin:builtin
2023-05-05T14:10:26.881Z error robot_server impl/resource_manager.go:566 error building resource resource rdk:service:vision/vis-stuff-detector model rdk:builtin:mlmodel error model "vis-stuff-detector" does not fulfill any method of the vision service. It is neither a detector, nor classifier, nor 3D segmenter
2023-05-05T14:10:26.861Z info robot_server.rdk:service:vision/vis-stuff-detector mlvision/ml_model.go:102 error turning turn ml model into a 3D segmenter model stuff-classifier error vision 3D segmenters from ML models are currently not supported
2023-05-05T14:10:26.861Z info robot_server.rdk:service:vision/vis-stuff-detector mlvision/ml_model.go:94 error turning ml model into a detector model stuff-classifier error Cannot use model as a detector
2023-05-05T14:10:26.765Z info robot_server.rdk:service:vision/vis-stuff-detector mlvision/ml_model.go:81 error turning ml model into a classifier model stuff-classifier error Cannot use model as a classifier
2023-05-05T14:10:26.636Z warn robot_server impl/local_robot.go:632 using deprecated robot constructor api rdk:service:vision model rdk:builtin:mlmodel

grave pier May 5, 2023, 2:12 PM

#

can you run:
pip show viam-sdk

#

and let me know the version of the sdk you are running?

gilded flare May 5, 2023, 2:12 PM

#

viampi@viampi:~ $ pip show viam-sdk
Name: viam-sdk
Version: 0.2.18
Summary: Viam Robotics Python SDK
Home-page: None
Author: Naveed
Author-email: [email protected]
License: Apache-2.0
Location: /home/viampi/.local/lib/python3.9/site-packages
Requires: googleapis-common-protos, Pillow, typing-extensions, grpclib, protobuf
Required-by:

grave pier May 5, 2023, 2:14 PM

#

hm, can copy/paste your raw JSON robot config from the app here?

gilded flare May 5, 2023, 2:14 PM

#

sure

#

📎 message.txt

grave pier May 5, 2023, 2:16 PM

#

it looks like you have both the new vision format and the old - can you remove the one that looks like:

      {
        "parameters": {
          "model_path": "/home/viampi/tutorial-openai-integration/lite-model_imagenet_mobilenet_v3_large_075_224_classification_5_metadata_1.tflite",
          "label_path": "/home/viampi/tutorial-openai-integration/labels.txt",
          "num_threads": 1
        },
        "type": "tflite_classifier",
        "name": "stuff_classifier"
      }
    ]
  },
  "name": "vision",
  "type": "vision"
},

gilded flare May 5, 2023, 2:28 PM

#

Modified it as you said

#

I reran the test

#

📎 message.txt

#

Here's the raw json again:

#

📎 message.txt

#

no output from show-detections

#

camera live view is functioning

#

I have viam support enable if that helps

grave pier May 5, 2023, 2:43 PM

#

So, you still have this, which is not needed:
{
"name": "vision",
"type": "vision",
"attributes": {
"register_models": []
}
},

#

And could cause issues potentially

#

And, your show-detections camera is not configured to use the classifier - looks like its attempting to use a color detector?

#

The Rosey tutorial in fact does not cover how to use a classifier as a transform camera, it just uses the classifier programmatically

#

But if you wanted to configure that, you could as per https://docs.viam.com/components/camera/transform/

Viam Documentation

Transform a Camera

Instructions for transforming a camera.

gilded flare May 5, 2023, 2:46 PM

#

I just removed that color detector service for now

grave pier May 5, 2023, 2:46 PM

#

Its not a bad idea to test the classifier with a transform camera

gilded flare May 5, 2023, 2:47 PM

#

grave pier So, you still have this, which is not needed: { "name": "vision", ...

removed this as well

gilded flare May 5, 2023, 2:47 PM

#

grave pier Its not a bad idea to test the classifier with a transform camera

sure, if you think it'll help

grave pier May 5, 2023, 2:49 PM

#

{
"name": "classifier-cam",
"type": "camera",
"attributes": {
"source": "cam",
"pipeline": [
{
"attributes": {
"confidence_threshold": 0.5,
"classifier_name": "vis-stuff-detector"
},
"type": "classifications"
}
]
},
"depends_on": [
"face-cam"
],
"model": "transform"
}

#

I think it would look like that

gilded flare May 5, 2023, 3:13 PM

#

Made the change

#

I notice if I remove an attribute from the builder mode, it's not removing it from the raw JSON

gilded flare May 5, 2023, 3:26 PM

#

grave pier { "name": "classifier-cam", "type": "camera", "attributes": { ...

Attribute Issues
Line 1: Problems loading reference 'http://definitions/transform-camera-12-emptyConfig.json': Unable to load schema from 'http://definitions/transform-camera-12-emptyConfig.json'. No schema request service available

#

I don't have the face cam component

#

2023-05-05T15:39:56.200Z error robot_server impl/resource_manager.go:566 error building resource resource rdk:service:vision/vis-stuff-detector model rdk:builtin:mlmodel error model "vis-stuff-detector" does not fulfill any method of the vision service. It is neither a detector, nor classifier, nor 3D segmenter
2023-05-05T15:39:56.185Z info robot_server.rdk:service:vision/vis-stuff-detector mlvision/ml_model.go:102 error turning turn ml model into a 3D segmenter model stuff-classifier error vision 3D segmenters from ML models are currently not supported
2023-05-05T15:39:56.185Z info robot_server.rdk:service:vision/vis-stuff-detector mlvision/ml_model.go:94 error turning ml model into a detector model stuff-classifier error Cannot use model as a detector
2023-05-05T15:39:56.101Z info robot_server.rdk:service:vision/vis-stuff-detector mlvision/ml_model.go:81 error turning ml model into a classifier model stuff-classifier error Cannot use model as a classifier
2023-05-05T15:39:55.977Z warn robot_server impl/local_robot.go:632 using deprecated robot constructor api rdk:service:vision model rdk:builtin:mlmodel

#

📎 message.txt

grave pier May 5, 2023, 5:24 PM

#

right sorry, on my robot its face-cam, on yours just 'cam'

gilded flare May 5, 2023, 5:36 PM

#

grave pier right sorry, on my robot its face-cam, on yours just 'cam'

all good. the config looks fine but it's still giving me the same errors

grave pier May 5, 2023, 5:37 PM

#

I think this error is a known issue but does not cause problems, so lets ignore it:
Line 1: Problems loading reference 'http://definitions/transform-camera-12-emptyConfig.json': Unable to load schema from 'http://definitions/transform-camera-12-emptyConfig.json'. No schema request service available

gilded flare May 5, 2023, 5:40 PM

#

ran the test.py script again for the current output

📎 message.txt

grave pier May 5, 2023, 5:42 PM

#

And viam-sdk is still 0.2.18? I ask because 0.2.19 changed VisionServiceClient to VisionClient (so the sample code would need to be updated for that, the app is not yet updated)

gilded flare May 5, 2023, 5:43 PM

#

yes, it's still 0.2.18

grave pier May 5, 2023, 5:44 PM

#

OK, let me try this locally and see if it works for me

gilded flare May 5, 2023, 5:44 PM

#

okay, thanks

#

should I update to 0.2.19? it sounds like that would fix the issues, right?

#

Successfully installed viam-sdk-0.2.19
viampi@viampi:~ $ python test.py
2023-05-05 13:46:28,367 ERROR viam (init.py:34) [ERROR] Uncaught exception
Traceback (most recent call last):
File "/home/viampi/test.py", line 13, in <module>
from viam.services.vision import VisionServiceClient
ImportError: cannot import name 'VisionServiceClient' from 'viam.services.vision' (/home/viampi/.local/lib/python3.9/site-packages/viam/services/vision/init.py)
Traceback (most recent call last):
File "/home/viampi/test.py", line 13, in <module>
from viam.services.vision import VisionServiceClient
ImportError: cannot import name 'VisionServiceClient' from 'viam.services.vision' (/home/viampi/.local/lib/python3.9/site-packages/viam/services/vision/init.py)

#

viampi@viampi:~/tutorial-openai-integration $ ./run.sh
pygame 2.1.2 (SDL 2.0.16, Python 3.9.2)
Hello from the pygame community. https://www.pygame.org/contribute.html
2023-05-05 13:52:52,881 ERROR viam (init.py:34) [ERROR] Uncaught exception
Traceback (most recent call last):
File "/home/viampi/tutorial-openai-integration/rosey.py", line 20, in <module>
from viam.services.vision import VisionServiceClient
ImportError: cannot import name 'VisionServiceClient' from 'viam.services.vision' (/home/viampi/.local/lib/python3.9/site-packages/viam/services/vision/init.py)
Traceback (most recent call last):
File "/home/viampi/tutorial-openai-integration/rosey.py", line 20, in <module>
from viam.services.vision import VisionServiceClient
ImportError: cannot import name 'VisionServiceClient' from 'viam.services.vision' (/home/viampi/.local/lib/python3.9/site-packages/viam/services/vision/init.py)

grave pier May 5, 2023, 7:11 PM

#

Yeah, you will need to update that to VisionClient as well if you are using 0.2.19 python SDK - just updated the tutorial repo with that change

gilded flare May 5, 2023, 7:43 PM

#

grave pier Yeah, you will need to update that to VisionClient as well if you are using 0.2....

📎 message.txt

grave pier May 5, 2023, 8:12 PM

#

ok, so that's now back to some issue apparently connecting to Viam

#

BTW, on the classifier - I too am able to replicate the problem setting up a classifier transform camera and am working with our Engineers on that

#

In the meantime, you could use a detector like the one found here: https://github.com/viam-labs/devrel-demos/tree/main/Light up bot

GitHub

devrel-demos/Light up bot at main · viam-labs/devrel-demos

Developer Relations demo scripts. Contribute to viam-labs/devrel-demos development by creating an account on GitHub.

#

But, that won't help with the connection issue you are seeing in what you most recently sent me

gilded flare May 5, 2023, 9:18 PM

#

viampi@viampi:~ $ python test.py
2023-05-05 17:17:53,815 ERROR viam (init.py:34) [ERROR] Uncaught exception
Traceback (most recent call last):
File "/home/viampi/test.py", line 13, in <module>
from viam.services.vision import VisionServiceClient
ImportError: cannot import name 'VisionServiceClient' from 'viam.services.vision' (/home/viampi/.local/lib/python3.9/site-packages/viam/services/vision/init.py)
Traceback (most recent call last):
File "/home/viampi/test.py", line 13, in <module>
from viam.services.vision import VisionServiceClient
ImportError: cannot import name 'VisionServiceClient' from 'viam.services.vision' (/home/viampi/.local/lib/python3.9/site-packages/viam/services/vision/init.py)

gilded flare May 5, 2023, 9:21 PM

#

grave pier BTW, on the classifier - I too am able to replicate the problem setting up a cla...

Is that the reason the vision service isn't working?

gilded flare May 8, 2023, 2:04 PM

#

Current run.sh output

📎 message.txt

#

@grave pier Is there anything I'm missing from my current config setup or anything else in general? I'm considering moving on from this project and trying something else in the meantime if we can't get it figured out. I'd like to continue figuring out if I do have something incorrect with my viam rover if that's the issue

grave pier May 9, 2023, 1:51 PM

#

OK, let's recap:

Running the rosey.py script on the pi is returning connection issues - seemingly can't connect to your robot
The classifier in this project is not working due to this issue: https://viam.atlassian.net/browse/RSDK-3009 . A detector could be used instead if you update your config and code as per this project: https://docs.viam.com/tutorials/projects/light-up/ - or you could alternately wait until RSDK-3009 is fixed
Does the python code from the code sample tab now run correctly from your pi? Note that if you are running Python SDK version 0.2.18 it will connect but then fail on the vision service call due to https://viam.atlassian.net/browse/APP-1779?search_id=3e26e431-0f56-4ef2-bb28-c1d21e571bdd (now fixed to be compatible with 0.2.19) - but at least we will know if connections to Viam work from your pi, and then we can focus on why the rosey.py script can't do the same

gilded flare May 9, 2023, 2:32 PM

#

The python code from the code sample does not run correctly. Looks like an issue with the vision service:
viampi@viampi:~ $ python test.py
2023-05-09 10:32:00,527 ERROR viam (init.py:34) [ERROR] Uncaught exception
Traceback (most recent call last):
File "/home/viampi/test.py", line 13, in <module>
from viam.services.vision import VisionServiceClient
ImportError: cannot import name 'VisionServiceClient' from 'viam.services.vision' (/home/viampi/.local/lib/python3.9/site-packages/viam/services/vision/init.py)
Traceback (most recent call last):
File "/home/viampi/test.py", line 13, in <module>
from viam.services.vision import VisionServiceClient
ImportError: cannot import name 'VisionServiceClient' from 'viam.services.vision' (/home/viampi/.local/lib/python3.9/site-packages/viam/services/vision/init.py)

#

I'll wait in the meantime for wait until RSDK-3009 is fixed unless I'm able to make time and update my config and code

sly solar Aug 3, 2023, 12:54 PM

#

@grave pier @gilded flare is there any further update on the issues in this thread. I tried to run rosey tutorial and faced same issues related ALSA, socket etc.

Yash Rai
5:11 PM (1 hour ago)
to pg

ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.bcm2835_headpho.pcm.front.0:CARD=0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM front
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib confmisc.c:1281:(snd_func_refer) Unable to find definition 'cards.bcm2835_headpho.pcm.surround51.0:CARD=0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory

Can anyone help here?

#

I am using
Name: viam-sdk
Version: 0.4.6

#Question about "Create an AI companion robot"

Traceback (most recent call last): File "/home/viampi/viam.py", line 3, in <module> from viam.robot.client import RobotClient File "/home/viampi/viam.py", line 3, in <module> from viam.robot.client import RobotClient ModuleNotFoundError: No module named 'viam.robot'; 'viam' is not a package

No worries, I tried to follow the steps in the doc

Traceback (most recent call last):
File "/home/viampi/viam.py", line 3, in <module>
from viam.robot.client import RobotClient
File "/home/viampi/viam.py", line 3, in <module>
from viam.robot.client import RobotClient
ModuleNotFoundError: No module named 'viam.robot'; 'viam' is not a package