UnicodeDecodeError когда пытаюсь запустить train.py
Я создал и подготовил виртуальную среду в anaconda для определения и распознавания обьектов (tensorflow object detection). Но на шаге 3 статьи по которой я следую я столкнулся с проблемой когда запускаю тренировку train.py:
python object_detection/legacy/train.py --logtostderr \ --train_dir=pack_detector/models/ssd_mobilenet_v1/train/ \ --pipeline_config_path=pack_detector/models/ssd_mobilenet_v1/ssd_mobilenet_v1_pack.config
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcd in position 112: invalid continuation byte
После попытки запустить train.py я попытался запустить model_main.py который как я прочитал запускает как train так и eval:
model_main_tf2.py пишет ту же ошибку
python object_detection/model_main.py --logtostderr \ --train_dir=pack_detector/models/ssd_mobilenet_v1/train/ \ --pipeline_config_path=pack_detector/models/ssd_mobilenet_v1/ssd_mobilenet_v1_pack.config
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 112: invalid continuation byte
все команды были запущены из директории \Documents\TensorFlow\models\research
Код который я использую (jupyter notebook)
это часть кода третьего шага статьи за которым я следую:
def class_text_to_int(row_label):
if row_label == 'pack':
return 1
else:
None
def split(df, group):
data = namedtuple('data', ['filename', 'object'])
gb = df.groupby(group)
return [data(filename, gb.get_group(x))
for filename, x in zip(gb.groups.keys(), gb.groups)]
def create_tf_example(group, path):
with tf.io.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid:
encoded_jpg = fid.read()
encoded_jpg_io = io.BytesIO(encoded_jpg)
image = Image.open(encoded_jpg_io)
width, height = image.size
filename = group.filename.encode('utf8')
image_format = b'jpg'
xmins = []
xmaxs = []
ymins = []
ymaxs = []
classes_text = []
classes = []
for index, row in group.object.iterrows():
xmins.append(row['xmin'] / width)
xmaxs.append(row['xmax'] / width)
ymins.append(row['ymin'] / height)
ymaxs.append(row['ymax'] / height)
classes_text.append(row['class'].encode('utf8'))
classes.append(class_text_to_int(row['class']))
tf_example = tf.train.Example(features=tf.train.Features(feature={
'image/height': dataset_util.int64_feature(height),
'image/width': dataset_util.int64_feature(width),
'image/filename': dataset_util.bytes_feature(filename),
'image/source_id': dataset_util.bytes_feature(filename),
'image/encoded': dataset_util.bytes_feature(encoded_jpg),
'image/format': dataset_util.bytes_feature(image_format),
'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
'image/object/class/label': dataset_util.int64_list_feature(classes),
}))
return tf_example
def convert_to_tf_records(images_path, examples, dst_file):
writer = tf.io.TFRecordWriter(dst_file)
grouped = split(examples, 'filename')
for group in grouped:
tf_example = create_tf_example(group, images_path)
writer.write(tf_example.SerializeToString())
writer.close()
tf_records(f'{cropped_path}train/', train_df, f'{detector_data_path}train.record')
convert_to_tf_records(f'{cropped_path}eval/', eval_df, f'{detector_data_path}eval.record')
#после этого я должен запустить тренировку train.py
Что я заметил так это то что проблема связана с декодингом .csv файлов, но в моем коде нечего не связано с этим файлом и его созданием, вместо этого, как я понял, создается некий dst_file. Также в других моделях я видел что после "return tf_example" идет следующий модуль, который как я считаю нужен и мне:
def main(_):
writer = tf.python_io.TFRecordWriter(FLAGS.output_path)
path = os.path.join(os.getcwd(), FLAGS.image_dir)
examples = pd.read_csv(FLAGS.csv_input)
grouped = split(examples, 'filename')
for group in grouped:
tf_example = create_tf_example(group, path)
writer.write(tf_example.SerializeToString())
writer.close()
output_path = os.path.join(os.getcwd(), FLAGS.output_path)
print('Successfully created the TFRecords: {}'.format(output_path))
if __name__ == '__main__':
tf.app.run()
Но если его просто вставить эту часть кода то выдаст ошибку:
if __name__ == '__main__':
^
IndentationError: unindent does not match any outer indentation level
Кстати, после установки object_detection-api я запускал model_builder_tf2_test.py с успешным выполнением поэтому модель, я считаю, рабочая:
C:\Users\Andrey\Documents\TensorFlow\models\research>python object_detection/builders/model_builder_tf2_test.py
2021-08-16 14:40:39.769707: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
Running tests under Python 3.9.6: C:\ProgramData\Anaconda3\envs\tf38v3\python.exe
[ RUN ] ModelBuilderTF2Test.test_create_center_net_deepmac
2021-08-16 14:40:43.808400: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021-08-16 14:40:44.041624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: NVIDIA GeForce MX250 computeCapability: 6.1
coreClock: 1.582GHz coreCount: 3 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 44.76GiB/s
2021-08-16 14:40:44.041888: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-08-16 14:40:44.051639: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-08-16 14:40:44.051792: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
...
[ OK ] ModelBuilderTF2Test.test_invalid_model_config_proto
[ RUN ] ModelBuilderTF2Test.test_invalid_second_stage_batch_size
INFO:tensorflow:time(__main__.ModelBuilderTF2Test.test_invalid_second_stage_batch_size): 0.0s
I0816 14:41:08.978053 11896 test_util.py:2102] time(__main__.ModelBuilderTF2Test.test_invalid_second_stage_batch_size): 0.0s
[ OK ] ModelBuilderTF2Test.test_invalid_second_stage_batch_size
[ RUN ] ModelBuilderTF2Test.test_session
[ SKIPPED ] ModelBuilderTF2Test.test_session
[ RUN ] ModelBuilderTF2Test.test_unknown_faster_rcnn_feature_extractor
INFO:tensorflow:time(__main__.ModelBuilderTF2Test.test_unknown_faster_rcnn_feature_extractor): 0.0s
I0816 14:41:08.993628 11896 test_util.py:2102] time(__main__.ModelBuilderTF2Test.test_unknown_faster_rcnn_feature_extractor): 0.0s
[ OK ] ModelBuilderTF2Test.test_unknown_faster_rcnn_feature_extractor
[ RUN ] ModelBuilderTF2Test.test_unknown_meta_architecture
INFO:tensorflow:time(__main__.ModelBuilderTF2Test.test_unknown_meta_architecture): 0.0s
I0816 14:41:08.993628 11896 test_util.py:2102] time(__main__.ModelBuilderTF2Test.test_unknown_meta_architecture): 0.0s
[ OK ] ModelBuilderTF2Test.test_unknown_meta_architecture
[ RUN ] ModelBuilderTF2Test.test_unknown_ssd_feature_extractor
INFO:tensorflow:time(__main__.ModelBuilderTF2Test.test_unknown_ssd_feature_extractor): 0.0s
I0816 14:41:08.993628 11896 test_util.py:2102] time(__main__.ModelBuilderTF2Test.test_unknown_ssd_feature_extractor): 0.0s
[ OK ] ModelBuilderTF2Test.test_unknown_ssd_feature_extractor
----------------------------------------------------------------------
Ran 24 tests in 25.205s
OK (skipped=1)
Версии пакетов
conda 4.10.1/ apache-beam 2.32.0rc1/ avro-python3 1.9.2.1/ cudatoolkit 11.3.1/ cudnn 8.2.1/ cython 0.29.24/ jupyter 1.0.0/ keras 2.4.3/ matplotlib 3.4.2/ notebook 6.4.3/ numpy 1.20.3/ object-detection 0.1/ opencv 4.5.1/ pandas 1.3.1/ pip 21.2.2/ protobuf 3.14.0/ pycocotools 2.0/ python 3.9.6/ scikit-learn 0.24.2/ six 1.16.0/ tensorflow (and gpu) 2.5.0/ tf-slim 1.1.0/
OS:Windows 10
GPU:MX250